Generating knowledge base articles from Jira tickets

Support logs an issue. Engineering fixes it. The ticket closes. Six months later somebody hits the same problem and there is no record of it anywhere a person would look — the reasoning lives in a comment thread on a resolved ticket that nobody will ever search.

Jira is a good tracker. It is a terrible knowledge base, because the two things want opposite structures: a tracker is organised by when, a knowledge base by what.

So I built a small service that reads a resolved ticket and produces a documentation draft from it. This is the design and the parts that turned out to be less obvious than they looked.

The shape of it

Four moving pieces, deliberately: a Jira client, a prompt, a store, and an approval step.

GET  /kb/jira/:ticketId    → fetch ticket + comments, return a draft
POST /kb/jira/finalize     → generate, mark complete, persist

Node and Express, Mongoose over MongoDB, the OpenAI chat completions API. Nothing exotic — the interesting decisions are not in the stack.

The draft is not the article

The first version wrote straight to the database. That was wrong, and obviously wrong the moment I read the output: an LLM given a ticket thread will confidently document a workaround that was discussed and rejected, because rejected ideas and accepted ones look identical in a comment thread.

So generation and persistence are separate calls. GET produces a draft that nobody has committed to. POST /finalize is a human saying yes, this is what happened. The isCompleted flag in the schema is the whole point of the schema:

const kbSchema = new mongoose.Schema(
  {
    ticketId:   { type: String, required: true, unique: true },
    sourceType: { type: String, enum: ["JIRA", "INTERCOM", "ZENDESK"], required: true },
    kb:         { type: String, required: true },
    isCompleted:{ type: Boolean, default: false },
  },
  { timestamps: true },
);

unique: true on ticketId matters more than it looks. Without it, a retried request produces a second article for the same ticket and now the knowledge base has two answers — which is worse than having none, because a reader cannot tell which one is current. The write is an upsert keyed on that field, so re-running is safe.

Comments are the content, not the description

The obvious implementation sends the ticket description to the model. The description is what someone thought the problem was before they understood it, so an article generated from it documents the wrong problem well.

The resolution lives in the comments. So the fetch is two calls, not one:

async function fetchJiraTicket(ticketId) {
  const [details, comments] = await Promise.all([
    fetch(`${jira.baseUrl}${ticketId}`, { headers }),
    fetch(`${jira.baseUrl}${ticketId}/comment`, { headers }),
  ]);
 
  if (!details.ok) {
    throw new Error(`Failed to fetch Jira ticket: ${details.statusText}`);
  }
 
  const issue = await details.json();
  const thread = await comments.json();
 
  return {
    source: "jira",
    id: ticketId,
    description: issue.fields.description,
    comments: thread.comments.map((c) => c.body),
  };
}

The model gets both, and the prompt says explicitly which one is the symptom and which is the diagnosis. That single distinction improved output quality more than any amount of prompt tuning.

sourceType exists for the connector I had not written yet

sourceType is an enum with three values and only one implementation. That looks like over-engineering, and it nearly was — but the shape of the problem is identical for Zendesk and Intercom, and the expensive part of adding them later would have been migrating a schema with no room for them.

The cost of the enum today is nothing. The cost of adding it to a populated collection later is a migration. That asymmetry is usually worth paying for up front, and it is the one piece of speculative design in here I would defend.

What I would do differently

The API is inconsistent, and I shipped it that way. GET /kb/jira/:ticketId reads an existing record rather than generating a draft, which is not what the endpoint name promises, and finalize fetches and generates internally rather than accepting the reviewed text. That means the human approval step I designed the system around is not actually enforced by the API — a caller can finalise without ever seeing a draft. The idea is right and the implementation does not honour it.

Linked tickets are ignored. Most real problems span several. An article built from one ticket in a chain documents a fragment and presents it as the whole.

There is no retrieval. With a few hundred articles, the useful question stops being "generate one" and becomes "is this already documented?" That needs embeddings and a vector store, and without it the collection will eventually accumulate three articles about the same thing.

The part worth keeping

Not the code — that was a weekend. The useful idea is that documentation debt is a capture problem rather than a writing problem. The information exists; it is already written down, by the people who solved the thing, at the moment they solved it. It is just sitting in a format nobody will ever read again.

Anything that moves it from there to somewhere searchable is worth more than a policy asking engineers to write documentation they will not write.