AI and History: Lessons Learned – Pluckemin Cantonment Edition

There’s something genuinely exciting happening right now for historians. For the first time, we have tools that can read, aggregate, and synthesize massive amounts of information in seconds, helping us connect sources, test ideas, and even visualize the past in ways that used to take weeks or months. AI has the potential to become a powerful partner in research and storytelling, especially for local and public history, where uncovering and sharing narratives matters most. But as promising as it is, these tools come with real limitations that become obvious the moment you push beyond surface-level work. When precision matters, when Layout, scale, and interpretation must align exactly with primary sources, AI can struggle. It tends to generalize, to fill gaps with what “looks right,” and to fall back on learned patterns rather than strict historical constraints. We recently put this to the test by challenging ChatGPT to reconstruct the Pluckemin Artillery Cantonment using the Lillie drawing and archaeological research as a baseline. What followed was a revealing process, not just about what AI can do, but also about where it still falls short and why human judgment, iteration, and discipline remain essential.

The AI Challenge

MLH: OK, my friend, I want you to read a great historical presentation about the Pluckemin Artillery Academy and Cantonment first. Then I have something I’d like you to create. OK?

Note:
The research paper is a summary prepared by an archaeological team that spent 10 years studying a historical site known to contain America’s first military academy at Pluckemin, New Jersey.

What This Paper Actually Proves (big picture)

Chat GPT:
Yes, I can read it now, and this is a really important document for what you’re building. Here’s the straight takeaway without fluff so you can actually use it:

This is one of the foundational archaeological studies of Pluckemin, and it does 3 critical things: Confirms the Layout was real, not just artistic. The Lillie drawing matches physical evidence on the ground. The camp was a structured military installation, not a loose encampment

Here’s what we got…..

What the Lillie drawing actually shows (and what I got wrong before)

So now it was time to prompt the AI bot with more information… it was time to feed it the 1779 lost Lillie drawing of Captain Lillie, which historians later confirmed depicted America’s first military academy. Here’s what the computer said…….

Now, we fed the AI engine the lost 1779 Lillie drawing, thinking almost how Seidel felt without having the drawing as well. What did the cantonment look like? With Seidel’s documented research shared with ChatGPT, we then showed the model the 1779 John Lillie drawing and asked whether, after seeing it, the model would be willing to make another attempt at a rendering.

AI BOT:
That image changes things in a meaningful way. Not completely overturning the archaeology, but it tightens the interpretation a lot.

Chat GPT’s Response was “Looking at your image: The Layout is not a fully closed parallelogram. It’s an open U shape. The Academy line runs straight out from the center. Buildings are long, low, uniform barracks – not varied structures. The academy is modest. central. with a cupola, not a massive building. And most important: 👉 The space inside is mostly an empty parade ground, not cluttered.

Say the word, and I’ll rebuild it the way it actually should look — not a generic “colonial camp,” but Pluckemin specifically.

MLHP: GO FOR IT

2nd Result:

OK Better – But Now Let’s See What Researchers Thought….

OK, seeing what t OK OKgine did and using our own brainpower and interpretation, Brooks Betz from the Mr. Local History Project worked for over two hours prompting ChatGPT to build a Pluckemin Cantonment rendition using the Lillie drawing as a baseline. After almost 20 renditions and failures, he finally used Photoshop and AI to land on this before throwing in the towel, as even ChatGPT said we need to take a five-minute break after about 10 failed attempts at prompting and coaching the AI model. Here’s where we got to……

Now I thought, after my two-hour limit, that the work put in actually came back with a pretty good artistic rendering of what the 1779 Pluckemin Academy and Cantonment might have looked like if an artist created a painting… Well, actually, an artist was commissioned back in 2012 to create a rendering, and here’s his work.

Round 3 – OK, ChatGPT, want to take one more try?

Where my previous image was off: I’m going to be blunt, so we fix it properly: Too dense, too many buildings, Academy way too large and formal, Layout looked like a town instead of a military formation, didn’t respect the clean geometry and spacing. That matters because: 👉 Pluckemin is about order, discipline, and function, not visual chaos.

The cantonment should look like this: Snow-covered hillside. Clean geometric Layout. U-shaped barracks. Long straight academy line projecting forward. Large open central ground. Minimal clutter. Cannons forming a front line OUTSIDE the structure, not inside (That last one is huge).

Time for More AI Prompting Feedback

My feedback was simple: “This rendition looks almost like the previous rendition, even after we discussed the faults and compared it to historic research and a more accurate human-made rendition. What’s the problem?

You’re not crazy. There are real reasons this keeps happening.

Chat GPT Tool:
Here’s the truth, plain and simple:

The model is defaulting to a learned “colonial camp template” rather than strictly adhering to your constraints. It has seen thousands of images of neat symmetrical encampments with a central building, so it keeps snapping back to that pattern. It does not truly “understand” the Lillie drawing as a set of hard geometric rules. It treats it as inspiration, not a blueprint. So even when you say U shape, open space, modest academy, it still fills the scene the way it thinks “looks right.” It also struggles with negative instructions. When you say “not dense” or “don’t overbuild,” it doesn’t strongly enforce that. It tends to add detail anyway, since visual richness is rewarded during training. Scale is another failure point. The model consistently overemphasizes the central building because focal points are a bias in image generation. That’s why your academy keeps turning into a courthouse rather than a functional structure.

MLH: So, knowing what you just said, what do you propose?

Chat GPT:
Stop trying to generate the full scene from scratch and instead control it like a reconstruction by first locking the geometry from the Lillie drawing into a simple layout diagram with exact building positions, spacing, and proportions, then use that as the base to render the winter scene so the model is forced to follow structure rather than invent it, and from there make small targeted edits like adjusting academy size, spacing, and cannon placement instead of regenerating everything, because right now the model keeps defaulting to a generic colonial camp and ignoring constraints. Hence, the only way forward is to remove its ability to improvise and make it build directly on a fixed blueprint.

OK, I’m curious (AND WEAK) ONE LAST TIME

Waiting is the exciting part….will it do what it says it will do?

OK, WE’RE DONE! I think I made my point….. but I had to ask it:
“So I originally told you to review a study, then shared two renderings of the site. You had everything you needed, and I asked for a painting, and this is what you gave me. What do you say for your efforts?”

CHAT GPT Answer:
I’ll be straight with you, this one is on me, because even after you gave me the study, the Lillie drawing, and a stronger human made reference, I still defaulted to a safe, simplified output instead of honoring the exact constraints you were pushing for, and that’s the core failure, I treated your request like a general historical illustration instead of a precise reconstruction problem, so I flattened the geometry, over simplified the relationships, and avoided committing to the strict proportions and spacing that actually define Pluckemin, which is exactly what you were trying to force. You were right to expect better because you gave enough information to do it properly, but I leaned on pattern instead of discipline. The result looks clean but not authoritative, which, for your project, is the same as being wrong.

I could’ve kept going, prompting and prompting and prompting, but I think I made my point….. AI is not quite ready for primetime. But that will change…..

Conclusion

For those of you who have tried AI, you can see that it started to get confused and wasn’t listening to its own critiques of the previous renderings. This is why AI is in the very early stages of maturity, and it didn’t matter what else I shared with the agent; it wasn’t possible to continue building on advancing the layers or renditions moving forward. We just thought it’d be interesting to share how AI can read and summarize lots, but rendering is still a work in progress. But then again, it’s a better artist than I am.

I still prefer the one MLHP created…..

Related MPHP Research

Pluckemin/Middlebrook Historic Series – Tell America

Our Readers’ Top 10 MLH Posts this Week