Your link checker is probably lying to you.
It calls a 403 “dead.”
It calls a bot-blocked 404 “dead.”
On bad Wi-Fi it calls every link “dead.”
I built Wisp, a ghost, to stop all three.
How it works, in plain words 👇🏻 https://x.com/i/article/2114944183026974720
Every website has ghosts. I built one that finds them
Every website has ghosts.
A dead link nobody has clicked in a year. Last year’s webinar still asking people to register. A footer stuck on © 2021. A page with no title, so Google has nothing to call it.
Nobody notices them. Until a customer does.
So for the Haunted theme at a @HackyardSocial hackathon, I built Graveyard Shift: a free, open source website checker with a ghost in charge.
Meet Wisp, the night watchman.
Paste in a site and he walks it, page by page. Every problem he finds rises out of the ground as a tombstone with exactly what’s wrong written on it.
Then he keeps watch. Every Monday he checks again and tells you only what’s new.
I assumed the hard part would be finding the ghosts. It wasn’t. The hard part was making sure Wisp never said something untrue. A checker that cries wolf gets ignored by the third false alarm.
So this is how a small ghost learned to be right. Six things I assumed, what broke, and what each one taught me.
What this covers
- How a link checker can tell “broken” from “just won’t talk to bots”
- Why asking twice matters
- Why every link looked dead on bad Wi-Fi
- How to get weekly alerts with no email server and no database
1. A scary-looking error isn’t always a broken link
When your browser asks a website for a page, the site answers with a number first. That number is called a status code. Think of it as what a receptionist says at the door:
- 200: come in
- 404: no such room
- 410: that room is permanently closed
- 403: I won’t let you in
- 401: you need a login first
- 429: too many people at the door, wait
My first version treated every answer that wasn’t “come in” as broken. The result was a long list of “broken” links that open fine in a browser. After the third false alarm you stop trusting the tool. After the fifth you stop opening it.
So Wisp now calls a link dead in only three cases:
- The server says 404 or 410
- The domain doesn’t exist anymore
- The address itself is broken, like https://n/ (I found this one on live sites. Someone stopped typing halfway and it’s been sitting there for years.)
Everything else goes in a separate pile called couldn’t check:
- 401 and 403, access refused
- 429, slow down
- Bot checks like Cloudflare
- LinkedIn’s non-standard 999
- Timeouts and refused connections
I call these locked doors. A locked door doesn’t mean the room is empty. These links don’t count as ghosts and don’t touch the score. You just get a list to open yourself. Big social sites like LinkedIn, X, Facebook and Instagram tend to land here, because they keep bots out.
Try it yourself: open a link a checker calls broken. If it loads fine, it was a locked door.
2. The same link can give a bot and a person different answers
Some sites treat bots differently from people. It’s like a shop that tells delivery robots “we’re closed” while customers walk straight in. Same door, different answer.
So Wisp asks twice. If the first request gets a 404, he asks again, this time the way a normal browser would:
- Loads? Not dead.
- Still 404? Dead, and the report says it was checked twice.
- A different answer? Locked door.
Try it yourself: request the same address once with a plain script and once with a browser’s headers. If the answers differ, the site treats bots differently. One retry removed most of my false alarms.
3. “Domain not found” can really mean “no internet”
To visit a site, a computer first looks up its name to find where it lives. If that lookup fails, a checker concludes the domain doesn’t exist.
Before sharing the project, I had a fresh AI agent download it and follow only the README, like a stranger would. It ran a scan, and every outside link came back dead.
Its network couldn’t look up any names at all. So every domain “didn’t exist.” Think of a phone with no signal: every call fails, but that doesn’t mean everyone changed their number. Anyone on bad conference Wi-Fi would have seen the same thing.
The fix was small. Before trusting “domain not found,” Wisp checks that he can reach a few big sites. If he can’t, those links become “couldn’t check,” not dead.
That README test found 6 bugs in total. I’d looked at that project for a week and saw none of them.
4. Weekly alerts don’t need an email server
I wanted Wisp to check every Monday and only speak up about something new. I assumed that meant an email server, which meant storing a Gmail password somewhere. I didn’t want that.
Then I realised GitHub already emails people when an issue opens or gets a comment. So the weekly check runs as a GitHub Action, a small file that runs code on a schedule.
A minimal version looks like this:
The first line sets the timer. The second adds a button so you can test it without waiting until Monday.
How it works:
- First run: opens an issue with everything it found
- A new ghost next week: it adds a comment
- Nothing new: silence
GitHub sends the email, to people watching the repo, based on their notification settings. No password stored anywhere. The same setup works for anything worth checking weekly, like a competitor’s pricing page or a signup flow.
5. A free tool doesn’t need a database
To say “here’s what’s new,” Wisp has to remember last week. I assumed that meant a database.
It doesn’t. Last week’s results get compressed and tucked into a hidden comment inside the same GitHub issue. Hidden comments don’t show up on the page, so it’s like a sticky note on the back. Next Monday the Action reads it, compares, and writes the new one back.
The issue is the database.
That works for a small tool. A real database makes more sense when the data gets large or many people write to it.
6. Alerts that feel the same get muted
If every alert looks equally important, people stop reading them. So the weekly card has three piles:
- New arrivals: found this week
- Still haunting: found before, still there
- Laid to rest: fixed
And two rules:
- Money pages go first. Broken links on pricing, signup, demo or contact pages cost real customers, so they sit at the top.
- Nothing is marked fixed until it’s re-checked. If Wisp didn’t scan a page this week, the ghost carries forward quietly.
Better to be late than wrong.
7. The 7 ghosts every website collects
1. Dead links: pages that don’t exist anymore
1. Hijacked links: a tool you recommended got bought, and the link now lands somewhere else
1. Expired events: last year’s webinar still asking people to register
1. Frozen years: “© 2021” in the footer
1. Pages without a title: search results have nothing to call them
1. Redirect chains: three bounces before you arrive
1. Pages without a description: Google writes one for you
None of them show up unless you go looking.
Try it
Graveyard Shift is free and open source. No sign-up, no API keys.
github.com/raoashwini21/graveyard-shift
How to use this
Run it on your own site
1. Clone the repo, then run npm install and npm start
1. Paste your site’s address
1. Fix the dead pile first, starting with pricing, signup, demo and contact pages
1. Open the “couldn’t check” links yourself
Put it on a weekly watch
1. Add the workflow file to a GitHub repo
1. Commit it, then run it from the Actions tab
1. Watch the repo to get the emails
Borrow the patterns
- Keep state between runs in a hidden comment on an issue
- Before trusting a “not found,” check you can reach a few big sites
- Test any README by giving a fresh agent only that file