I asked my golf coach a simple question: my irons keep curving left, what should I do?
It told me how to fix my putting.
The coach was a small language model I had spent a month and a half fine-tuning so it could live inside PinTrace, the golf app I built. The answer was confident, well formatted, and completely about the wrong club. That moment sums up most of what I learned from the project, and it is why PinTrace ships today without the model I worked so hard on.
Why I wanted a model on the phone
PinTrace started as a tool for me and the people around me. Renting a server for a handful of users felt like overkill, and I liked the idea that a player’s golf data would never have to leave their phone. Around the same time I found an open-source project on GitHub that walked through training a small model from scratch. It made the whole thing look approachable. So I asked the obvious question: why not fine-tune my own model and run it on the device?
I set the direction and the tasks; AI coding agents did most of the hands-on work, preparing data, running training jobs on my Mac and building the evaluations. A stronger model (GPT-5.6 Terra) wrote example answers for the small model to learn from, and another (GPT-5.6 Sol) reviewed them. The models I trained were small, 0.8 and 2 billion parameters, small enough to fit on an iPhone.
Lesson 1: read the answers, not the average
The first scores looked good. Then I started reading the answers one by one, and the model was inventing my statistics.
This happened three times, and each time the numbers said everything was fine. An automatic checker designed to catch made-up personal statistics let 16 of 17 fabricated ones through. One test arm reported zero unsupported claims while 15 of its 34 answers contained invented numbers. Every one of these was found by a person reading the output, never by a score.
Later we tested the automatic grader itself. On the questions it had been tuned on, it agreed with a human reader 74% of the time. On fresh questions it had never seen, that dropped to 50%, a coin flip. So we changed the process: a fresh set of test questions was written and frozen before every round of training, and nobody looked at it until the end. We even caught training examples that were quietly paraphrasing test questions, and threw them out.
Lesson 2: fix the input before blaming the model
The most surprising discovery was that many failures were not the model’s fault at all.
The app handed the model the player’s statistics as raw labels, like gir=57%. GIR means greens in regulation, the share of holes where you reach the green in the expected number of strokes, so higher is better. The model, guessing at the label, translated it as a miss rate and confidently told players the opposite of the truth. When we rewrote the input in plain language, “greens in regulation (GIR): 57%”, that kind of error dropped from 12 cases to 1, without training anything.
Then we tried removing the statistics from the prompt entirely, expecting cleaner answers. Instead, asked about its own fairway percentage, the model invented one: “68% over the last 90 days,” when the real number was 75%. The data I had thought of as clutter was actually what kept the model honest.
Even the safety net needed checking. Once, the model correctly pointed out that a club-head speed of 210 mph is physically impossible. Our validator rejected that answer, because it mentioned club-head speed without the exact wording the rule expected. The checker was throwing away the right answer.
Lesson 3: a model can learn the voice without learning the game
After the input was cleaned up, the remaining problem really was the model. So we trained the larger one to answer golf questions directly, using 163 carefully reviewed example answers.
On a fresh test it got 14 of 20 golf questions wrong. It told players to strike a punch shot “at the top of the backswing.” It sounded exactly like a coach, the tone, the structure, the confidence, but the golf was wrong. Training had taught it how a coach talks, not what a coach knows.
What did work was giving the small model a narrower job. Instead of answering, the 0.8B model only decided what kind of question it was looking at. At that it beat the bigger model, 36 of 38 against 26 of 38. The answers themselves came from a knowledge base of 207 golf facts, each checked against a reputable source.
The decision
At the start of October I had a version of the app ready for review with a model inside it. I pulled it.
It was not an easy call. I had spent a month and a half on fine-tuning, and part of me wanted to ship it just to have something to show for that time. But I could not honestly say the model was reliable, and a coach that is confidently wrong is worse than no coach.
PinTrace 1.0 shipped without any local model. Everything the app decides on its own runs on transparent rules. AI answers happen only when a player asks, through a cloud model, with their data summarised on the phone first. Before a model is offered, it has to pass the same kind of evaluation I learned to run on my own. One free model invented numbers in 6 of 13 test documents and was rejected. The one we use today invented none.
What changed in how I judge ideas
This project changed how I evaluate a plan.
Before, when I talked an idea through with an AI and it laid out a complete-looking path to build it, I tended to assume the rest was just time and effort: follow the steps and it will work. Actually building it showed me something different. Problems nobody had anticipated kept appearing, and more fixing and more training did not necessarily get the result where I wanted it.
Now I treat a plausible plan as a hypothesis, not a schedule. I decide up front how I will know whether it worked, freeze the test before I start, and read the actual output instead of trusting a single number. And I am more willing to stop. A month and a half of fine-tuning did not give me a model I could ship, but it gave me a way of working that every part of PinTrace now depends on.
That is also why I want to study the systems side of AI. Building a model is the exciting part. Knowing whether it actually works, under real constraints, is the part I want to get good at.