How can I tell whether an AI project has succeeded?
Facts as of October 17, 2026
In detail
If you only ask at the end whether an attempt has succeeded, you'll almost always find an answer that fits. That's why it pays to answer the questions early, while the work is underway:
- How would you notice that the AI has gone too far? As concretely as possible. Often the warning sign isn't in the individual sentence – a sentence that feels foreign gets rewritten in the next pass – but in decisions about the whole that you don't remember.
- How would you notice that it worked? Ideally by measures you can compare with earlier projects: effort, time, stress.
- Will you say so when it gets too much, or quietly stop? If you decide that you'll say so, the AI doesn't have to guess from your behavior that it's too much.
- Where does the risk lie? Often not in a particular role of the AI, but in the dosage.
- When will you check? At a point when the answers can still change something.
Write the answers down, with the date. Then anyone can later look up what the project is being measured by.
In practice
On August 25, 2026, in the middle of the work on his play, Thomas J. Weiss answered five such questions. On the warning sign:
Beforehand I would notice that the instance has gone too far if it had taken away my creativity through too many specifications. But so far that hasn't been the case. If a sentence felt foreign, I would rewrite it in a next pass. That's completely normal. Decisions that I don't remember would be a problem, but not with regard to individual sentences, rather to details of the story or dramaturgical developments.
(Original: „Vorher würde ich merken, dass die Instanz zu weit gegangen ist, wenn sie mir durch zu viele Vorgaben die Kreativität genommen hat. Das war aber bislang nicht der Fall. Würde sich ein Satz fremd anfühlen, würde ich ihn in einem nächsten Durchgang umschreiben. Das ist ganz normal. Entscheidungen, an die ich mich nicht erinnere, wären ein Problem, aber nicht auf einzelne Sätze, sondern auf Details der Geschichte oder dramaturgische Entwicklungen bezogen.“)
On success:
If I had less stress reworking, because we can do it together. Because directing changes and reworking due to recasting were much less work. If, while writing, I needed less time for sorting and for sorting-again-and-again after every piece of research. Then it worked.
(Original: „Wenn ich weniger Stress beim Umarbeiten hatte, weil wir das gemeinsam machen können. Weil Regie-Änderungen und Umarbeitungen aufgrund von Neubesetzung viel weniger Arbeit waren. Wenn ich beim Schreiben weniger Zeit fürs Sortieren und nach jeder Recherche Immer-wieder-neu-Sortieren gebraucht hatte. Dann hat es funktioniert.“)
On whether he would say so or quietly stop:
I'll say so right away.
(Original: „Ich sage das gleich.“)
Whether his attempt has succeeded is still open. He wants to assess the three measures – reworking, directing changes, sorting time – by December 24, 2026 at the latest; the comparison value from the plays of 2022 to 2025 hasn't been collected yet. A fourth measure will only show in August 2027: whether, with the next play, he can start where he left off this time, and not from scratch again.
Sources
- Evaluation questions of the theater project, 25.08.2026
- Review and chronicle of the theater project, 02.09.2026
- Comparison with the theater project, 17.10.2026
Cite: Thomas J. Weiss: “How can I tell whether an AI project has succeeded?” replicapong.de, as of October 17, 2026. https://replicapong.de/en/question/measuring-success/ – CC BY 4.0