Misplaced Trust on The Internet | LUMP Depot

You can buy GitHub stars. A few bucks gets you a thousand of them, delivered overnight from a pile of throwaway accounts. I keep coming back to that fact because of what a star is supposed to mean. It's supposed to mean a real developer looked at this repo and thought it was worth keeping. Thousands of stars is supposed to mean thousands of those little votes of confidence.

Except the signal that's meant to say "lots of real people vetted this" can also say "one person had a credit card." And it cascades. A blog post links the starred repo as proof the library is solid. A build pipeline pulls it in because it's "popular." Somebody's threat model says "well, it has 12k stars, it's probably not malware." Each layer trusts the layer below it without ever checking the bottom.

I drew a diagram about it

I sketched this out to settle the argument in my own head.

A hand-drawn diagram titled The Problem with Compound Liars, with three horizontal bands labeled Low Trust, Medium Trust, and High Trust. Boxes sit at different heights. The Website says Trust Me Bro sits at low trust. An arrow goes from a box pointing to a security-sounding site that also says Trust Me Bro up to Security Site is Fake. A box for a score based on number of reviews points up to Reviews are Fake. The website uses encryption sits at medium trust pointing to The verifier is fake or insecure. The website puts proof on the blockchain points up to The proof is real but the data was a lie.

The thing the diagram is really about is that trust is recursive. Every proof system leans on some other proof system. A liar can build a compound lie by wrapping a false claim inside layers of verification that all look independent but aren't.

You can walk the ladder:

  • "Trust me bro" is the floor. Nobody believes it.
  • "A security site says trust me" is higher, but only if the security site is itself trustworthy. Plenty of them are just a logo and a green checkmark someone bought.
  • "Thousands of reviews say it's good" is higher still, but only if the reviews are real. See also: the stars.
  • "It's on the blockchain" feels highest of all, but only buys you immutability, not honesty.

A proof proves existence, not truth

Here's the line I keep wanting to tattoo on people's foreheads. A proof can show that a statement existed, was signed, was recorded, or was never altered. It cannot, by itself, show that the original statement was true.

The blockchain version of this confuses everyone, so it's worth pulling apart. People mix up two completely different things:

Integrity: this data wasn't modified.

Truth: this data accurately describes reality.

A blockchain is fantastic at the first one and almost entirely useless for the second, unless you also trust the process that produced the data in the first place. Put a lie on the blockchain and you get a permanent, tamper-proof, beautifully signed lie. The chain did its job perfectly. It just wasn't the job you thought it was.

It's a graph, not a ladder

The vertical scale in my diagram is a little white lie I told myself to make it readable. Trust isn't actually a ladder where each rung is strictly higher than the last.

Take "the website uses encryption." That doesn't make the claims on the website more true. It proves secure transport, nothing else. A scam site with a valid HTTPS cert is still a scam site, now with a padlock icon to make you feel better about it.

And the reverse trips people up too. "The verifier is fake or insecure" doesn't automatically make the claim false. It just kills that one verification path. The claim might still check out somewhere else.

So the honest picture isn't a ladder at all. It's a graph of dependencies. Every trust claim has upstream assumptions hanging off it, and every one of those assumptions is its own little door an attacker can try.

The weakest link is usually the data, not the math

This is the part that credential systems, review systems, reputation scores, and blockchain projects all keep tripping over. The weak link is almost never the cryptography. It's the data source feeding into it.

Think about a diploma. It can be cryptographically signed. A blockchain can timestamp it. Three different verifiers can confirm the signature is valid. Airtight, end to end.

But if the institution that issued it was a diploma mill running out of a P.O. box, every layer of that stack faithfully preserves a worthless credential. The math is perfect. The input was garbage. That's the ceiling on any proof system, no matter how clever the proof is.

Which brings it back to the stars. The git history is cryptographically perfect. Every commit hashes to the next, the whole chain verifies, nobody tampered with anything. None of that makes the 12,000 stars real. The proof was guarding the wrong door.

And now we're feeding it to the machine

Here's the part that actually keeps me up at night. We used to be the ones reading the stars and the reviews and the security badges, and at least a human can get a bad feeling and dig deeper. Now we're training models on all of it.

An LLM learns by reading the internet at a scale no person ever could. It sees the repo with 12,000 bought stars and the blog posts that cite it as a trusted library and the SEO pages that exist only to launder that lie into looking like consensus. It has no way to check the bottom of the stack either. It just sees the pile and learns that the pile is tall.

So the compound lie doesn't stay an opinion you can argue with. It gets baked into the weights as a fact. The model can't un-believe it later, because there's no original source to go back to, just a frozen statistical average of everyone who repeated it. Garbage in, garbage faithfully preserved, exactly like the signed diploma from the fake school.

At that point asking an LLM what to trust is about as reliable as buying the first thing Amazon recommends. It looks authoritative, it's wrapped in the confidence of a system that processed a billion signals, and underneath it's just whatever got promoted the hardest. The recommendation isn't the truth. It's the output of whoever gamed the inputs best.

So check the bottom of the stack

The compound liar's whole trick is that we trust the verification chain without examining each dependency in it. We see "signed, recorded, verified, popular" and stop reading. The fix isn't more cryptography. It's the boring, annoying habit of asking where the original claim came from, and whether you'd trust that source if it were standing in front of you with no padlock, no checkmark, and no stars. That habit matters more than ever now that we've handed the reading to machines that can't form a bad feeling.

tl;dr: You can buy GitHub stars, and that's a tidy demo of how trust works online. Every proof leans on another proof, so a compound liar stacks fake ones until the pile looks tall. Proofs are great at integrity (this wasn't changed) and useless at truth (this was true to begin with). It's a dependency graph, not a ladder, and the weakest link is almost always the data source, not the math. And once an LLM trains on all those baked-in lies, trusting its answer is about as smart as buying the first thing Amazon recommends. Check the bottom of the stack.