Some Sunday Sentiments on Science and Substack Search Sucking

(Pretty Poorly Proofed)

I want Substack to work for me as a reader.

I’m not overly concerned about it working for me as writer, but that’s a story for another time.

And yes, this is a bit of a rant, but it’s also substantive…

I posted a couple of notes today about how substack’s not working for me as a reader, and that made me think — perhaps erroneously — that I should make a post about what’s not working and why (in theory).

The substack recommendation feed rarely serves anything that I’m interested in. On the rare occasions when it’s at least trying, like it’s actually noticed what I’m into… it just fails to nail the assignment.

Yet, many of the posts that are recommended to me (that I’m not interested in) are singing the praises of substack’s content delivery:

“You’ve shown me so much wonderful writing OMG thank u substack I finally found my community!” (The Live-Laugh- LoveStack)

Yes, ‘the live-laugh-lovestack’ is something that I just made up… but you probably had to think about it.

Although, I didn’t search to see if that exists, and there’s a good reason why:

Substack search is a piece of shit.

All the posts from grateful readers who’d found what they are interested in made me think that maybe the mismatch between algorithm and me is my fault, because I haven’t been interacting with posts.

(Of course, substack’s not serving me posts I want to interact with, so chicken, meet egg. Egg, chicken.)

I tried using search to find posts to interact with, and my god is that worthless.

No advanced options. No boolean. Doesn’t search all terms by default. It seems to be ranking results based on the same busted-ass algorithm that I’m trying to ‘fix’.

Look, we (the human race) figured out how to search databases, oh, 40 or 50 years ago. Developers, please, I beg of you:

Stop ranking results by some assumed relevance to me, and just rank by relevance to the search terms.

Seriously. I’m a big boy. I know how to search effectively. Been doing that for a minute. Stop making all searches work (i.e. not) like Facebook Marketplace.

I can’t even figure out a search string that would bring up my own content if someone happened to be interested in that. And I love my own stuff, so clearly their relevance-based search isn’t working.

The one slightly promising substack that search did return… I’m pretty sure is a sophisticated AI user.

They seem to have a workflow consisting of:

1. find a popular YouTube video

2. have AI analyze, summarize, and create commentary

3. edit that into a polished and seemingly high-level essay…

…which is actually bland, relatively shallow, and has a near complete absence of any point of view beyond the most predictable one.

It’s freshman A-student writing, no doubt. But that’s not actually good writing, it’s just lacking obvious flaws that every decent writer would try (but frequently fail because they’re on a deadline) to avoid.

I’m not linking that because it’s possible the content is human all the way down. In which case, I’d feel bad for claiming otherwise… but the criticism stands.

After a few (failed) searches, I gave up and went back to the feed and things were… different.

Not better, just different. Like, focused, if someone still didn’t know how to do that properly. Like the holiday gifts you get from people who don’t really know you.

There was one substack that caught my eye, because it was about automotive (my field, roughly).

It describes itself as created by an alternating group of authors with different voices and unique perspectives on automotive for “consumers, enthusiasts and industry insiders.” (I stan the Oxford comma, but will try not to hold that against them.)

Rad!

So, I checked them out.

The ‘different voices’ and ‘unique perspectives’ belong to six middle-aged white dudes.

They’ve all worked for the same industry-standard magazines-turned-websites.

Those outlets are utterly indistinguishable from each other. They’re tired. They’re predictable. They’re… “bland, relatively shallow, and has a near complete absence of any point of view beyond the most predictable one.”

Those sites consist of writing that could have been produced by AI, even if it isn’t. Increasingly it is, which is probably why those six dudes are no longer writing for those outlets.

I’m a middle-aged white dude, so you’d think — as does the substack algorithm — that I’d be the target demo for that shit.

But because I’m a middle-aged white dude I already know what a middle-aged white dude’s perspective is (as do most non-middle aged, non-white, non-dudes, but that’s a much broader topic).

I don’t need to read more like that. I’ve been reading that for decades. It’s not interesting, fresh, unpredictable, or new…

Ah-ha! That’s what I want: interesting, fresh, unpredictable, new!

Do you hear me, substack algorithm?!

(I assure you, it doesn’t.)

But…. I want ‘interesting, fresh, unpredictable, new’ in my own field. That’s a bit trickier.

The problem with algorithms and AI is that, without a shit-load of training, they struggle to identify what’s relevant to an expert in a particular field.

I’m not trying to jock myself here, just using the ‘10,000 hour rule’: An expert is someone who’s practiced something for 10,000 hours. That makes me an expert on automotive media.

By the way, the ‘10,000 hour rule’ is often credited to Malcolm Gladwell. But his version is a pop-psych oversimplification of what was actually shown through the research of psychologist K. Anders Ericsson who said, “this rule is wrong in several ways.”

So, you know, it’s the usual: Science is actually complicated so don’t read a headline and act like you now know some shit.

Anyway.

Experts typically get around the problem of algorithms/AI with… search.

Beginners have so many gaps in their knowledge that searching for what ‘fits’ in those holes is nearly impossible for them. One of the gaps in their knowledge is the knowledge of how to fill those gaps. But a decent algorithm can do that (just).

Experts are aware of the gaps in their knowledge and what would fill them. This is actually what makes most science possible. Experts know exactly what to search for to fill in those gaps. If they didn’t, they couldn’t design relevant experiments. Experiments are just ways of searching the world, after all.

Unfortunately, for that gap-knowledge to work when searching a database (which substack is, for all intents and purposes here), that requires a search function that’s not using the same god damn fucking non-expert algorithm!!

Whew.

I think I know why substack search is like this, though.

Search is an afterthought. It’s something which is mostly just relevant to readers and, unfortunately:

Substack is not for readers.

Just look at the about page.

If you scroll down, it’s entirely about what joining substack as a creator can do for you. All the tools they’ve built for creators. How much they will promote your content. And of course how much of that sweet, sweet creator money you will make.

Note that you may get a redirect if you’re logged into substack, so you may not see this version of the about page if you aren’t browsing anonymously. This is what is shown to those who are not on the platform, which is why it matters.

There is not a single sentence on the substack about-page that says, “Hey! Substack is a great place to read/listen/watch!”

It’s entirely about substack being a great place to create content, with nothing about it being even a good place for consuming that content. (Well, fair enough, because that’s roughly how I feel about it right now.)

As a content-consumer… why the fuck should I care about substack?

Genuine question.

Because it doesn’t seem to have a way to deliver content I actually care about.

And as a creator, if the consumers aren’t here, aren’t being actively pursued by the platform, and aren’t being engaged by the content delivery systems when they are here… why would I want to create on this platform?

Also a genuine question.

---

Alright, that’s actually a lot more than I planned to write on this topic. If you stuck it out… Thanks? Congratulations?

Joking aside, I am grateful for everyone who reads what I write.

And if substack is giving you good recommendations (i.e. this post, maybe?)… Please let me know the how, what, why, and how long it took to get that to work for you!

One last thing:

Next post will be back to ‘normal’, explaining how YouTube has fucked creators by pushing Shorts.

See you then!