This page was written by one or more anonymous contributors.
Title: Why I'm Not A Postrat Date: 2021-12-11
What Is A Postrat, Anyway?
As I've written about before, 'postrat' isn't really any one thing. There's no core canon or explicit ideology to refute, just a loose association of cliques and geeks adjacent to the defunct ruins of LessWrong Rationality who Talk About Stuff™. In fact, described like that it's not obvious why someone would want to write about them at all. This is by explicit design, as we'll go into later the postrats deliberately obfuscate their ideas, purposes, and social presentation so that describing them is like nailing jello to a wall. Most of my purpose for writing is they get a lot of attention (collectively they have hundreds of thousands of followers on Twitter) while saying things that seem plainly wrong to me. Furthermore critiques of postrat are few and far between, the ones that do exist are so bad they are actively embarrassing to their authors:
Some people be like "I feel like this fails to adequately summarize the postrat thesis" and I be like "WHAT thesis". They're way too post to have a shared thesis. That's exactly the kind of rat they're post.
— Eliezer Yudkowsky (@ESYudkowsky) December 19, 2020
Contra Yudkowsky I think there is a postrat thesis, but getting at it is appropriately difficult. There's a sense in which stating the thesis of postrat is to have refuted it. Attempting that now in the introduction would be especially foolish. Instead in this essay I will respond to three major ideas that, interpreted broadly, I think capture 90% of what postrat is 'about'. These are:
Various paradoxes surrounding 'rational irrationality', or the idea that instrumental rationality implies that epistemic rationality is irrational.
Objections to the existence of universal laws in thought, physics, etc.
Buddhist or quasi-Buddhist objections to the negativity and utilitarian aesthetics of the LessWrong rationalist worldview.
Rather than dig deep into any of these right away, I think it would be useful to set up the warrant for postrat's critique of LessWrong rationality first. As usual it is premised on a large pile of undigested updates, realizations, and unannounced shifts in consensus. And also as usual to my knowledge these haven't been summarized before in one place, a task once again left to myself.
So lets do one more LastRat post, for old times sake.
What Was LessWrong Rationality Supposed To Do?
Probably the easiest way to contextualize 'postrationality' is to take a trip back to 2011, when The Sequences were still fresh and the "rationality community" was in its early founding period. It was around then that the goals of the movement solidified, and we can understand where postrationality comes from by looking at how those goals failed or lost relevance.
After visiting the New York chapter Eliezer Yudkowsky wrote them a letter to say what he thinks of their progress and where he expects them to run into issues. In this letter he articulates the vision he has for the young rationality movement:
“Stay on track toward what?” you ask, and my best shot at describing the vision is as follows:
“Through rationality we shall become awesome, and invent and test systematic methods for making people awesome, and plot to optimize everything in sight, and the more fun we have the more people will want to join us.”
(That last part is something I only realized was Really Important after visiting New York.)
Michael Vassar says he’s worried that you might be losing track of the “rationality” and “world optimization” parts of this—that people might be wondering what sort of benefit “rationality” delivers as opposed to, say, paleo dieting. (Note—less worried about this now that I’ve met the group in person. -EY.)
I admit that the original Less Wrong sequences did not heavily emphasize the benefits for everyday life (as opposed to solving ridiculously hard scientific problems). This is something I plan to fix with my forthcoming book—along with the problem where the key info is scattered over six hundred blog posts that only truly dedicated people and/or serious procrastinators can find the time to read.
(bolding mine)
This concept of 'systematic winning' is what Yudkowsky ultimately centered his stated philosophy on. It is a concept I've previously criticized as incoherent but for now we'll run with it and see where it goes. The basic idea is that when Spock says he won't do the straightforward thing because "that would be illogical", the rationalist is meant to bonk Spock over the head, then point out that he is not maximizing his expected utility by focusing on aesthetics over winning. LessWrong rationality then is economic rationality, not aesthetic or logical purity. This conflict is brought up explicitly in The Sequences with an example from decision theory called Newcomb's Problem.
In Newcomb's problem an entity that can predict the future, called 'Omega' offers you a choice involving two boxes. In one box is a million dollars, in the second box a hundred dollars. If Omega predicts that (and it is very good at prediction) you will only take the box with the million dollars, you will open it to find a million dollars. If it predicts you will take both boxes the million dollar box will be empty. This contradicts traditional Spock-ian notions of decision theory because it says that the best way to maximize your utility is to not take the 2nd box even when you know it contains an extra hundred dollars and Omega has no more influence on the result once it's there. Intuitively you should be able to walk over, pick up both boxes, and get an extra hundred dollars on top of your million.
Yudkowsky points at the sort of philosopher who insists they would take both boxes because it would be irrational not to as the prototype of anti-economically-rational behavior:
It is, I would say, a general principle of rationality—indeed, part of how I define rationality—that you never end up envying someone else’s mere choices. You might envy someone their genes, if Omega rewards genes, or if the genes give you a generally happier disposition. But Rachel, above, envies Irene her choice, and only her choice, irrespective of what algorithm Irene used to make it. Rachel wishes just that she had a disposition to choose differently.
You shouldn’t claim to be more rational than someone and simultaneously envy them their choice—only their choice. Just do the act you envy.
I keep trying to say that rationality is the winning-Way, but causal decision theorists insist that taking both boxes is what really wins, because you can’t possibly do better by leaving $1000 on the table… even though the single-boxers leave the experiment with more money. Be careful of this sort of argument, any time you find yourself defining the “winner” as someone other than the agent who is currently smiling from on top of a giant heap of utility.
So when Yudkowsky says he wants the New York LessWrongians to invent and test systematic ways for making people awesome, he means as much as anything else that he wants them to systematically win at life. Say for example, figure out the best fitness training regimen and then find a way to consistently get yourselves to follow it.
Raising The Sanity Waterline
The other really big idea is raising the sanity waterline. Yudkowsky points out that over time people have become less "black cats are a witch's familiar" in their beliefs and more materialist-y. That there are general principles of thinking like Occam's Razor which would make someone less likely to believe in woo and magic. Or as he puts it:
Suppose we have a scientist who’s still religious, either full-blown scriptural-religion, or in the sense of tossing around vague casual endorsements of “spirituality”.
We now know this person is not applying any technical, explicit understanding of... * ...what constitutes evidence and why; * ...Occam’s Razor; * ...how the above two rules derive from the lawful and causal operation of minds as mapping engines, and do not switch off when you talk about tooth fairies; * ...how to tell the difference between a real answer and a curiosity-stopper; * ...how to rethink matters for themselves instead of just repeating things they heard; * ...certain general trends of science over the last three thousand years; * ...the difficult arts of actually updating on new evidence and relinquishing old beliefs; * ...epistemology 101; * ...self-honesty 201; * ...etcetera etcetera etcetera and so on.
So a lot of Yudkowsky's hope was that an effective rationality movement could push people farther away from black cat familiars and closer to physics. It would do this not just by directly arguing against nonsense the way New Atheists do, but by popularizing habits of thought which make it harder for the nonsense to survive as living ideas. A further side effect of this is that people would become better at recognizing and acting on good ideas, both increasing the likelihood that the world survives the 21st century and that the radical opportunities our civilizations are sleeping on actually get pursued. He writes in his final Sequences post:
In the end, these 21 months were wagered upon your active response, not your passive possession of truth. Even those of you who are not moved to care about the thing that I protect, may still be moved to align their activities with Seasteading, or the Methuselah Foundation, or Givewell, or cryonics. And, perhaps, tell them that we sent you, so that they know that this “rationality” business is working to their favor; so that maybe they will say a thing or two about rationality. And someone who encounters the craft there, will read on it further, and eventually donate professionally specialized hours (in one form or another) to the Singularity Institute. What goes around comes around, I hope.
To the extent that you don’t feel moved to do anything in particular—even include a link in your signature, or link to apposite posts now and then in your blog—then I have lost my gamble of these 21 months. Or I have lost that part of the stakes which was about you and decision processes similar to you. (No, that doesn’t mean you should seize this opportunity to post about how I lost my gamble with you. You should know better by now, if you want any rationalist cause to get anything done ever, whether or not you are a part of it.)
As for why Earth needs rational activists in particular—I hope that by now this has become clear. In this fragile Earth there are many tasks which are underserved by irrational altruists. Scope insensitivity and the purchase of moral satisfaction leads people to donate to puppy pounds as easily as existential risk prevention; circular altruism prevents them from going so far as to multiply utilons by probabilities; unsocialized in basic economics, they see money as a dirty thing inferior to volunteering unspecialized labor; they try to purchase warm fuzzies and status and utilons all at the same time; they feel nervous outside of conventional groups and follow the first thought that associates to “charity”...
And with all that in mind we're ready to start talking about postrat!
The Case Against LessWrong Rationality
Well, sort of.
We're ready to start talking about why the core thesii of LW, "raise the sanity waterline" and "learn to win at everything" are nonviable. In the process we'll begin stumbling into postrat observations and ideas.
Epistemic Rationality and Instrumental Rationality Do Not Cohere
LessWrongers eventually came to distinguish between two kinds of rationality: epistemic rationality which deals with knowing what is true, and instrumental rationality which deals with the 'systematic winning' stuff discussed previously. Given that one of Yudkowsky's core examples of instrumental rationality is taking an 'irrational' action in the service of winning, it's not a difficult observation to make that 'irrationality' can be very economically rational. In Yudkowsky's view however epistemic rationality and instrumental rationality are tightly linked, to the point where the former almost implies the latter. But this is only gestured at, naively we should expect it to be true but it's not like we have some theoretical model that promises us it is. And in fact it's not an uncommon belief to think that caring a great deal about epistemic rationality is unfit.
Sonya Mann has written a formulation of a classic postrat proof that instrumental rationality recommends against epistemic rationality:
Granted, there is a well-established edge in believing facts. But only those facts which you act upon; only in those circumstances where if you bet wrong, you'll experience a consequence. In any other domain, rationally you should believe whatever it is most advantageous to believe, both in terms of how the beliefs affect your behavior and in terms of the signaling value to be wrung from them.
Someone who perfectly calibrates which of their beliefs to base on facts versus fashion can be expected to outcompete someone else who tries to use facts as the basis for every belief. After all, various different narratives will fit the same set of facts.
If you start with the facts and ignore the prevailing narrative, or are ignorant of it, then you might reason your way into a place where you're not allowed to go.
Mustn't do that.
Namely that the 'most rational' thing to do in the general sense is not to believe what is true, but what is fashionable. Scott Alexander has written about something similar to this as the concept of anti-induction. A thing is anti-inductive when it responds to your attempts to understand it by becoming something else. Take for example strategy in a game like Chess or Pokemon. Once a strategy becomes known as 'the best strategy' an opponent using it is predictable and eventually the strategy stops working as its weaknesses are ruthlessly exploited by savvy players. The stock market works in a similar way. There's no "one weird tip" to beat the stock market because any strategy that works will eventually drive up the price of its underlying assets until no more outsized risk-adjusted return is possible. Gaining deep insight into these adversarial environments changes the nature of those environments. They're not natural-philosophy shaped and trying to study them the way you might study a species of bird or a function of the body would be a fatal mistake.
Take even something that might seem totally objective like an Olympian fitness training regimen. Your muscles don't get harder to train as you learn to train them. Acrobatics and sprints don't learn to resist your superior athletic techniques over time. But your competition very much gets stiffer as new methods are learned. The level of performance that would have made you an Olympic champion in the 1950's wouldn't even get you a spot in the competition today. So lets say you ground yourself up to this seemingly impossible standard and then added a few new tricks on top to take the gold. Your competition would catch on and soon enough everyone else would be doing the same things, it wouldn't be a systematic, repeatable way to win, in adversarial games those aren't possible even in principle. One video by Summoning Salt on the insane lengths speedrunners will go to for shaving fractions of a second off their game should cure you of that idea.
Now you might think to yourself "okay, so avoid the things where adversarial dynamics exist", but society itself is an adversarial dynamic! Social epistemology is anti-inductive and often has perverse game theoretic dynamics that make seeking the truth personally costly. Everyone is familiar with the story of Galileo. Acting righteous in a perverse environment is antisocial. In such an environment socially naive rationality is a liability, not an asset. "Rationality is winning" is not just one of the biggest epistemic mistakes in The Sequences, but probably its biggest marketing mistake too. It's extropy-as-prosperity-gospel and when the prosperity is not forthcoming people quite reasonably abandon it for something with more immediate returns.
But this logic can be extended further with another of Scott Alexander's posts on the conflict vs. mistake theory of politics. The post can be summarized as the idea that some people think their political opponents are mistaken and some people think their political opponents are intrinsically socioeconomically opposed to them (i.e. rich vs. poor). In the world where your political opponents are mistaken discourse and dialogue make a lot of sense, they're how you discover preferences and where the problems lie. In the world where you and your opponents are intrinsically opposed there isn't much to discuss, since at the end of the day you have to fight each other anyway. There it makes sense for arguments to be soldiers, it's not like you can actually talk people out of their class interests. Debate and argument in political matters are just propaganda for getting your side to turn out.
This paranoid orientation reaches its zenith in the concept of 'legibility'. Legibility is the idea that states have a preference for structures they can tax and monitor. A farm with evenly spaced grids of crop monocultures is much easier to measure and tax than a more productive forest densely interwoven with symbiotic flora and fauna. Governance itself is the act of making things legible so they can be taxed and monitored, so resisting authoritarian state power means obfuscating how things work. This can be slightly generalized out to a Boydian confusion-conflict model that says you should deliberately make yourself hard to model so opponents find it difficult to compute their gametree with respect to you.
Lets say you're the sort of person to read Yudkowsky's Sequences. You enjoy them because you broadly share his values and priorities. Learning all this then puts you in quite a conundrum. On the one hand you care deeply about human freedom and a positive future. On the other hand obstinately insisting the adversarial environment you're in is actually a random (mistaken) one attracts parasites. Culture warriors parasite your descriptive language for solving problems to make propaganda and act out their class interests. When a deadly pandemic sweeps through your country you find that your utilitarian suggestion to institute travel restrictions becomes the basis for wild authoritarian power grabs. All utilitarian measures you suggest which increase authoritarian state power are implemented immediately, all measures which would solve the problem (domestic mask production, vaccinations, PPE for healthcare workers, testing for the virus, etc) are endlessly delayed. At some point the 'incompetence' becomes so overwhelming that even you, optimistic and trusting person you are no longer believe that 'random incompetence' can explain what seems like a concerted conspiracy to prolong the pandemic.
You stop sharing your helpful suggestions, knowing that they will only be used as a backdrop to harm others by the unscrupulous. You obfuscate your social presentation, becoming quieter and more ironic. It becomes hard to tell when you are serious and when you are joking. Others who have seen what you have seen begin to gravitate towards you. You never discuss exactly what you are going to do about all this, but you know an open confrontation with the corruption in your society will go nowhere. So you wait in the shadows, watching carefully for an opportunity to strike.
Raising The Sanity Waterline and Information Warfare
I have this very blurry memory from high school of the first time a malicious viral Internet sensation forced itself into my life. On most mornings we spent the first 30 minutes of school in 'advisory' where we could discuss any academic issues we were facing with a faculty member. But that morning the faculty called every student into the auditorium for a special announcement, where they intended to show us the Kony 2012 documentary. At the mere mention of it multiple students stood up and started shouting back at the stage, I can't even remember what they were shouting, just that they were begging them not to play the film and insisting it was a hoax. They played it of course, and I hadn't heard of Kony 2012 so my experience of this was as a total outsider. A surreal scene of what was functionally an Internet meme determining the itinerary of what was supposed to be a vetted educational institution as the students themselves leaped to their feet in protest of the memetic cancer. I felt bewildered and confused, a birth pang of what was to come for the rest of the 2010's.
Without systematized winning as a carrot, it's difficult to believe that raising the sanity waterline in the way envisioned by Sequences!Eliezer is really possible. But the situation became hopeless in the current media landscape. The new currency of the Internet was attention, and the best emotional appeal for getting it was rage. As Scott Alexander would later write the best way to get a frothy rage going isn't to say reasonable things about outrageous things, there are an uncountable number of outrages we ignore on a daily basis. Instead the optimal profile for viral content is outrageous speech in opposition to outrageous subjects. Feminism becomes about killing men, socialism about worshipping Stalin, tradition about throwing people out of helicopters, liberty about social darwinism, love about huge harems and anonymous orgies. Every value or cause imaginable becomes a twisted parody of itself, gaining power and followers in direct proportion to its ugliness. Hysterical mobs of mindkilled cyborgs cheer on the liminal space between comedy and brutality, adhering closer and closer to ironic beliefs until they are sincere. Society becomes irony poisoned and the patient begins to die.
In that wasteland it is no longer just a matter of rationality not providing outsized returns, 'society' (or its shambling corpse) sternly punishes clear and reasonable thinking. Those who allow themselves to be possessed by demons wield greater and greater power in society while those who resist become marginal. Worse than marginal, they become prey. The demons heap their scorn and abuse on anyone who is not like them, and lead witch hunts to systematically eliminate those who dare to point out that their behavior has become perverse. It is difficult to find someone who believed the things Eliezer Yudkowsky believed in 2010 who is not at least somewhat traumatized by the time the decade ends in 2020.
As it turns out there is much to hide from, and an esoteric meaning of 'postrationalist' begins to reveal itself.
Raising The Sanity Waterline and Our Epistemic Crisis
Even if social media was never invented we would still be in the throes of a deep epistemic crisis. One brought on by the failure to replicate many studies previously considered established science. The Control Group Is Out Of Control is easily one of Scott's best posts, likely in the top 5. It goes beyond summarizing the replication crisis (at the time thought confined only to psychology) and explains how the entire enterprise of scientific investigation has become busted for the questions we want it to answer. His case study of a meta-analysis 'proving' the existence of psychic powers is one of those gems of rationalist writing worth quoting in its entire length:
Bem, Tressoldi, Rabeyron, and Duggan (2014), full text available for download at the top bar of the link above, is parapsychology’s way of saying “thanks but no thanks” to the idea of a more rigorous scientific paradigm making them quietly wither away.
You might remember Bem as the prestigious establishment psychologist who decided to try his hand at parapsychology and to his and everyone else’s surprise got positive results. Everyone had a lot of criticisms, some of which were very very good, and the study failed replication several times. Case closed, right?
Earlier this month Bem came back with a meta-analysis of ninety replications from tens of thousands of participants in thirty three laboratories in fourteen countries confirming his original finding, p < 1.2 × 10−10, Bayes factor 7.4 × 109, funnel plot beautifully symmetrical, p-hacking curve nice and right-skewed, Orwin fail-safe n of 559, et cetera, et cetera, et cetera.
By my count, Bem follows all of the commandments except [6] and [10]. He apologizes for not using pre-registration, but says it’s okay because the studies were exact replications of a previous study that makes it impossible for an unsavory researcher to change the parameters halfway through and does pretty much the same thing. And he apologizes for the small effect size but points out that some effect sizes are legitimately very small, this is no smaller than a lot of other commonly-accepted results, and that a high enough p-value ought to make up for a low effect size.
This is far better than the average meta-analysis. Bem has always been pretty careful and this is no exception. Yet its conclusion is that psychic powers exist.
So – once again – what now, motherfuckers?
'What now?' indeed. Stepping back for a minute we can note that our phenomena are largely conserved. The shocking development here is not that physics has been overturned or our fundamental ontologies need to change. Planes still fly and computers still beep boop, up is still up and down is still down. Rather what we have discovered is that our ability to investigate subtle phenomena with science is fundamentally blocked by the complexity of managing (and enforcing good faith in) scientific study. Studies can 'prove' this or that small effect (like the slight case of psychic powers) largely independently of the actual existence of those effects. This might not seem like a big deal until you realize that 20th century physics has been entirely about control of the nth decimal point. Our ability to precisely investigate marginalia has been the great source of our recent power, and that power is now waning.
The replication crisis has so far been treated as an internal dispute in academia, an esoteric game of insider baseball for scientists to fix by stacking more statistical minutia on top of their methods. What makes Scott's post great is that he sees beyond this to the truth: The 'replication crisis' is an epistemic crisis of full blown proportions with no known solution. Future historians may see it as the epoch making problem of our time. It is of no less importance than the Enlightenment realization that the previous standards of proof used up until that era, expert testimony and large numbers of eyewitnesses, could often speak to the existence of plain absurdities. In 1762 Rousseau noted of Antoine Calmet's scholarly work on vampires:
If there is in this world a well-attested account, it is that of vampires. Nothing is lacking: official reports, affidavits of well-known people, of surgeons, of priests, of magistrates; the judicial proof is most complete. And with all that, who is there who believes in vampires?
A passage shockingly similar in structure to Scott's thoughts on psychic powers.
Why I Am Not A Postrationalist
Most eternalist systems promise vastness: becoming part of something far greater than yourself. Theism connects you with God, who is incomparably superior to the entirety of His Creation, and gives you a role to play in His Plan. Scientism’s sales pitches invoke the unimaginable scale of the astronomical universe, and promise complete understanding of it with physical laws that—like theism’s Biblical laws—are absolute truths. Political theories explain the millennial sweep of human history, and the global struggle for justice, in terms of immutable laws of society, and they offer you a starring role in determining whether future generations will live in an oppressive dystopia or glorious utopia.
— David Chapman, Vaster Than Ideology
Right about now is when I'm supposed to give you the stars speech. You know, the one where I talk about how there's a vast future ahead of us if we can just get off this planet and how incomprehensibly huge the light cone we can colonize is. The one where I talk about how we can become protean demigods liberated from death on much longer timescales than an ordinary human lifespan. The part where I discuss how your monkey brain is tuned to suffer for reasons that don't make sense anymore and we can give even the lowest member of society superhuman equanimity and bliss. It's where I'm supposed to sweet talk you with poetry and try to seduce you with my passion.
But lets be honest, the stars speech would be a waste of breath on you.
If you're a postrat you've almost certainly heard it before and no longer find it compelling. The stars speech is a promise, it doesn't prove anything. It is an enticement to explore and learn more, but the value of seducing people with poetry and passion is pretty low overall. You can seduce people to Christianity or Islam or Hinduism with poetry and passion. Even truly absurd and bizarre cults can manage this. If that's all your belief is based on, aesthetics, the barrier separating you from any of these other creeds is thin indeed.
I'm sure you don't believe in promises anymore. If you're here, you know, here and you related to all that stuff I just wrote you are almost by definition traumatized. Going from a worldview where you think that when you try to help you're helping to one where all your attempts to help are ran through a filter of malevolence is blackpill, it's super blackpill. Before I can write the rest of this post we need to get over this "rats are traumatized but I'm going to be okay" cope. Any realistic description of the rigorous postrat worldview is just as blackpilled and traumatized as X-Risk longtermists or whatever. 'Trauma' as a frame is problematic because it implies the way forward is to hit the undo button, that whatever separated you from normality is something that could in principle be 'healed' by going back, that to become a normie would be a miracle cure. If you internalize bad news, why shouldn't it affect your behavior? I would hope you don't really believe you were better off not knowing, that it was better when you were in the bad situation and unaware of the situation.
But honestly? A disappointing number of people do believe they're better off not knowing. One of my formative adolescent experiences was getting to watch the 2008 housing crash up close, things got very bad and I observed adults much older than me cope with the situation. Almost all of them seemed to have at least one major problem they were refusing to acknowledge, even as their lack of acknowledgement was hurting them and everyone else around them. I distinctly remember a moment where I saw two people mutually drop the ball for each other because they were so afraid to see things as they were. I vowed then and there that I would never live like this, that I would go out of my way to see the plain reality of things even if it hurt. Epistemic squalor became as intolerable to me as environmental dysfunction. Much is made of the ways you can hurt others by being insufficiently aware of yourself, but just as common is hurting others by lacking the bravery to mentally live in the real world. It's one thing to have your time wasted by "the law of attraction" when you're an atomized individual who largely hurts only themselves with delusion, it is much darker when you have a newborn infant and their future is dependent on you making good decisions about your career and income.
On The Ethical Desirability Of A Non-Inductive Universe
Both the Illuminati and MJ12 believe that the most intelligent or enlightened human being will inevitably gain power, ultimately seizing the eye in the pyramid and creating the world for everyone else. Kind of secularized version of natural law.
— Nicolette DuClare, Deus Ex (2000)
The narrative presented by James Scott in Seeing Like A State contrasts local geographic knowledge with imperial monitoring. Take for example this description of farming practices in Scott Alexander's review:
Ye olde organically-evolved peasant villages tended to be complicated confusions of everybody trying to raise fifty different crops at the same time on awkwardly shaped cramped parcels of land. Modern scientific rationalists came up with a better idea: giant collective mechanized farms growing purpose-bred high-yield crops and arranged in (say it with me) evenly-spaced rectangular grids. Yet for some reason, these giant collective farms had lower yields per acre than the old traditional methods, and wherever they arose famine and mass starvation followed. And again, for some reason governments continued to push the more “modern” methods, whether it was socialist collectives in the USSR, big agricultural corporations in the US, or sprawling banana plantations in the Third World.
&ellip;
Why did all of these schemes fail? And more importantly, why were they celebrated, rewarded, and continued, even when the fact of their failure became too obvious to ignore? Scott gives a two part answer.
The first part of the story is High Modernism, an aesthetic taste masquerading as a scientific philosophy. The High Modernists claimed to be about figuring out the most efficient and high-tech way of doing things, but most of them knew little relevant math or science and were basically just LARPing being rational by placing things in evenly-spaced rectangular grids.
But the High Modernists were pawns in service of a deeper motive: the centralized state wanted the world to be “legible”, ie arranged in a way that made it easy to monitor and control. An intact forest might be more productive than an evenly-spaced rectangular grid of Norway spruce, but it was harder to legislate rules for, or assess taxes on.
This frame is set up to give us the impression that induction and universal principles are an invasive force. In this story 'legibility' is an imperial tool violating a superior, preexisting local order. It serves authority and rulers at the expense of their subjects. In fact all three of the vignettes Scott gives in his review are of this pattern: A geographically local system that works is imposed upon by an outside imperial power, making the locals worse off to strengthen state control. It's a compelling narrative that primes us to think of inference itself as a villain, encouraging ironic and obsfuscatory behavior to thwart surveillance.
The sleight of hand trick that makes it work is the contrast between geographies. Once we move away from that to localism in ideas and concepts, towards specialization, it becomes obvious that a world without induction can only be ruthlessly authoritarian in its social structure. Without inductive principles there is no such thing as epistemic leverage, the only way to know more about a subject is linear knowledge acquisition. In this universe knowledge is taxonomic. Consider the job of the historian, whose role is to know the specific sequence of events that occurred among people at a particular place and time. There are no general principles he can appeal to, he has read a set of books and learned the lore from a set of masters and his knowledge cannot go much beyond them. In the world without induction all fields are like history: There is only an ever growing catalogue of empirical observations to take note of, and nobody has knowledge of anything which has not been observed and written down.
In a world without epistemic leverage, specialists rule with an iron fist. The average laborer cannot devote hours and hours a day to studying the esoterica that keeps his civilization going, so she will always be the servant of those who can. A highly produced "public" health expert cannot be questioned, their decades of study apply only to their specialization and knowledge from outside that bubble cannot pierce it. The public is at the total mercy of little sects and insider circles. If they are found to be corrupt there is no recourse for the layman: she can only learn at the same glacial linear rate as her corrupt overlords, ensuring their corruption does not cost them too much in virtue. Only the well heeled and elite can have any contribution to discourse, because they have the time and resources to soak in enough of the crucial esoteric wisdom to be worth speaking to. Everyone else is literally inferior and knows it, elites beat them freely when they are so much as inconvenienced by their presence.
This is of course not a description of an alternative universe at all, but the condition of man in preindustrial agricultural society.
Universal Principles and Laws of Thought/Physics/etc Do Not Exist
There's a frustrating motte and bailey argument made by postrats when it comes to physics and metaphysics. The easily defensible argument(s) comes in several flavors like "Bayesian methods do not provide a complete account of how to think well" and "there is no rational theory of everything you can apply to win at life". These relatively uncontroversial statements are then used to suggest a proof-by-pattern match to much less defensible ones like "there is no computable function which approximates the physics of our reality" and "there is only local knowledge, global insight is a modernist scam to constrain thinking to the legible". To help dissolve this (likely deliberate) confusion I'll start by stating the obvious: A complete theory of epistemology would be closely related, if not equivalent to, a complete theory of how to build AGI. Accepting that humans represent general intelligence and no known formal method does, it is necessarily always going to be the case that any system of epistemology used by humans is going to be less powerful in one way or another than the brain that is running it. All known formal methods, even if they provide local guarantees or insight into a particular domain, are less general than a human brain. So of course a human using them will need to interject and make bridges and cover for their flaws. Duh.
If that was all there was to it I'd probably end the discussion there. "This is confused, we don't actually have a disagreement on the things you say we do, bye."
But then you encounter some batshit insane stuff like this and it's on like Donkey Kong!
Unfortunately one of the claims at issue here is about what Eliezer Yudkowsky really believes, which is exactly the kind of tedious nerd shit that's boring to write and probably boring to read. So to be merciful I'll try to keep this short: I say that it's more or less uncontroversial that there is no rational theory of decision making/thought/everything (whether that's Bayesian statistics or anything else) a human can implement in their brainware to reliably win at life. And the question is if Sequences!Eliezer would agree with this.
I don't know, honestly.
I'm not trying to be coy either, Yudkowsky sends some really mixed messages on this in his writing. If someone skimmed or read a bit of it and came out with the impression that he believes one thing or the other I wouldn't be mad, because depending on which bit you read you could get a totally different impression. For example, in the post ['Initiation Ceremony'])https://www.readthesequences.com/Initiation-Ceremony) he writes a ha-ha-only-serious parody(?) of his own viewpoint which strongly implies Eliezer thinks Bayes is an actual theory-of-everything for human cognition. This post is referenced later in his discussion of Einstein, whose method of physics was to use reason to infer what the struture of the universe must be for it to be inductive:
And the people in that alternate Earth would say, “The final equation was simple, but there was no way you could possibly know to arrive at that answer from just the perihelion precession of Mercury. It takes many, many additional experiments. You must have measured time running slower in a stronger gravitational field; you must have measured light bending around stars. Only then can you imagine our unified theory of ethereal gravitation. No, not even a perfect Bayesian superintelligence could know it!—for there would be many ad-hoc theories consistent with the perihelion precession alone.”
In our world, Einstein didn’t even use the perihelion precession of Mercury, except for verification of his answer produced by other means. Einstein sat down in his armchair, and thought about how he would have designed the universe, to look the way he thought a universe should look—for example, that you shouldn’t ought to be able to distinguish yourself accelerating in one direction, from the rest of the universe accelerating in the other direction.
And Einstein executed the whole long (multi-year!) chain of armchair reasoning, without making any mistakes that would have required further experimental evidence to pull him back on track.
Even Jeffreyssai would be grudgingly impressed. Though he would still ding Einstein a point or two for the cosmological constant. (I don’t ding Einstein for the cosmological constant because it later turned out to be real. I try to avoid criticizing people on occasions where they are right.)
(emphasis mine)
This entire discussion occurs in the context of Yudkowsky trying to explain to the reader why the version of science they were taught in 5th grade based solely on empiricism is not and cannot be how science works on any reasonable human timescale. He's trying to point at some of the metacognitive machinery which is necessary to make that naive scientific method work.
I say 'metacognitive' rather than 'metarational' because Yudkowsky isn't actually stepping out of systems yet. And why should he? I would hope we're all aware by now that paradox and contradiction are intrinsic to any mildly complex formal system, but the only reason why this was ever a shocking insight at all is that systems fail in related-enough ways to make it seem like they could be reconciled if you squint. So it stands to reason that there will exist systematic (though obviously themselves imperfect) mitigations to various recurring problems in systems. There's something hilarious to me about the idea of supposed meta-rational masters throwing their hands up and abandoning systems the minute they run into their first choice between greater and lesser paradox. You're sitting there freaking out about implicit infinite sums in decision theory while the average person can't even reliably make sure their hypothesis space sums to 1. Which problem would you rather have: An esoteric borderline-sophist issue that humans are very good at reliably avoiding using their intuition, or an endemic problem affecting nearly all machinery of thought that human intuition leans into and amplifies?
If you think you want the latter I hate to break this to you but you are not 'meta rational', you are a romantic.
And if we read to the end of his thoughts Yudkowsky certainly seems like he's finally stepping out of systems, writing:
I have striven for a long time now to convey, pass on, share a piece of the strange thing I touched, which seems to me so precious. And I’m not sure that I ever said the central rhythm into words. Maybe you can find it by listening to the notes. I can say these words but not the rule that generates them, or the rule behind the rule; one can only hope that by using the ideas, perhaps, similar machinery might be born inside you. Remember that all human efforts at learning arcana slide by default into passwords, hymns, and floating assertions.
(bolding mine)
Or a bit before that in another of his tongue-in-cheek vignettes involving fictional Bayesian monks:
“Don’t,” Jeffreyssai said. There was real pain in it. “Believe me, it hurts me more than it hurts you.” He might have been looking at them; or at something far away, or long ago. “I don’t know exactly what roads may lie before you—but yes, I know you’re not ready. I know I’m sending you out unprepared. I know that everything I taught you is incomplete. That what I said is not what you heard. I know that I left out the one most important thing. That the rhythm at the center of everything is missing and astray. I know that you will harm yourself in the course of trying to use what I taught; so that I, personally, will have shaped, in some fashion unknown to me, the very knife that will cut you…
“… that’s the hell of being a teacher, you see,” Jeffreyssai said. Something grim flickered in his expression. “Nonetheless, you’re done. Finished, for now. What lies between you and mastery is not another classroom. We are fortunate, or perhaps not fortunate, that the road to power does not wend only through lecture halls. Or the quest would be boring to the bitter end. Still, I cannot teach you; and so it is a moot point whether I would. There is no master here whose art is all inherited. Even the beisutsukai have never discovered how to teach certain things; it is possible that such an event has been prohibited. And so you can only arrive at mastery by using to the fullest the techniques you have already learned, facing challenges and apprehending them, mastering the tools you have been taught until they shatter in your hands—”
Jeffreyssai’s eyes were hard, as though steeled in acceptance of unwelcome news.
“—and you are left in the midst of wreckage absolute. That is where I, your teacher, am sending you. You are not beisutsukai masters. I cannot create masters. I cannot even come close. Go, then, and fail.”
So if nothing else Sequences!Eliezer seems to be aware that he is not handing out some kind of magic formula for life, or making decisions, or anything like that. But does he believe one is possible in principle? It's hard to say. I'm not convinced that the Eliezer Yudkowsky I can ask about this on Twitter is meaningfully the same person as the one who wrote The Sequences for the purposes of that question. My earlier discussion of his vision for the rationality community implies there's a really good chance he actually might, which would be naive and tragic. But I think it's also quite likely that if you'd pressed him on it at the time, he'd eventually say more or less what I'd say:
There are useful systems of thought which apply to various domains, some of these systems are proven to be optimal or close to optimal within their domain. Some of these systems have very wide and very general domains. You should use them whenever it makes sense. Sometimes, these systems will encounter pathological cases or suggest their use outside of their domain, at this point there are various meta-systems you can apply to patch and bridge the gaps between your systems. However these are also, in the end, systems, and will have their own failure modes and domain limitations beyond which your only choice is to use your head the old fashioned way: imprecisely, intuitively, backed by a very powerful sapien cognitive engine which is correct way more often than chance but also frequently wrong for reasons that are not always easy to debug.
Furthermore, while this might make it sound like those cases are rare, actually lots and lots of daily stuff, if not the average thing you do on a given day should not involve systems. The environment is pretty tractable for sapien hardware and you should shut up and let it do its job.
&ellip; Or we could just go read Yudkowsky's later debate with Chapman in which he says much the same thing:
Not to commit the fallacy of the golden mean or anything, but the two viewpoints are both metatools in the metatoolbox, as it were. You’re better off if you can use both in ways that depend on context and circumstance, rather than insisting that only toolbox reasoning is the universally best context-insensitive metaway to think.
If that’s not putting the point too sharply.
Thinking in terms of Law is often useful. You just have to be careful to understand the context and the caveats: when is the right time to think in Law, how to think in Law, and what type of problems call for Lawful thinking.
Which is not the same as saying that every Law has exceptions. Thermodynamics still holds even at times, like playing tennis, when it’s not a good time to be thinking about thermodynamics. If you thought that every Law had exceptions because it wasn’t always useful to think about that Law, you’d be rejecting the metatool of Law entirely, and thinking in toolbox terms at a time when it wasn’t useful to do so.
In the final analysis I guess my answer to "Does Eliezer Yudkowsky unreasonably believe in a royal road to mental perfection in unaltered humans?" is no.
But Do Universal Laws Exist?
{...}
As a sanity check we can start by putting an upper bound on the disunity of nature by making the anthropic-cartesian observation that nature has to be unified enough for us to exist in it. If nature was actually local to the point where it wasn't possible to make inferences that hold from one environment to another, we probably wouldn't exist. We can further infer that it has to be unified enough for us to experience building a metal can with explosives attached which is capable of being slingshotted up out of our planets orbit and over to a nearby celestial object while both bodies are in motion. We also have to contend with the observation that recent technological advancements have been clustered, as though there were general insights we were gaining that allow us to usher forth many new inventions based on similar sets of techniques. Even if we hypothesize that the reward structure of our universe is evenly distributed and we're just putting in more resources to get the rewards, we have to account for the sudden increase in resources available to put towards extracting more rewards. In other words, we have to contend with the fact that from a naive perspective our universe behaves in a pretty darn unified way even within the ordinary human experience.
All of this is suggestive, but do universal laws exist?
Yes.
Of course they do.
Usually when making this point people pick arithmetic because it's simple and almost every reader of this essay knows how to do it. You say "two and two make four" and BAM universal law demonstrated, science and reason win, debate over right? Well not exactly. See it's not that this particular example is wrong, even David Chapman admits this is true, it's just that it is so cliche that it has lost all meaning as anything other than a way to say which side of the argument you're on. But there's also another problem.
Numbers are actually confusing.
Like, really confusing. What is "two" of something, what is a "two", why do we have a sense of "twoness" which can be added to another thing with property "twoness" to get "fourness"? It's actually a very abstract concept relying on a fair amount of innate machinery and if you start talking about it explicitly people get tripped up. In the Principia Mathematica, which sets out to show a mathematics without assumptions through the use of assumptions so small its authors can no longer see them Russell and Whitehead finally manage to prove 1 + 1 = 2 on page 86 of the second volume. This is a good hint that while concepts like "two" are ontologically basic for human minds (which need no training to understand natural numbers up to four) they're actually an inductive bias, not something with an intrinsically low conceptual complexity.
So instead I'd like to demonstrate with something I expect most readers struggling on this point are not familiar with, Shannon's information theory. Even though information theory was only discovered in the 20th century it's conceptually pretty simple, arguably about as simple as arithmetic, and has the benefit of being so alien to ancestral human experience that we can only understand it explicitly, preventing our intuitions from getting in the way. It's also a universal law, the sort of thing that should be true in any universe we have any business thinking about (any universes we don't have any business thinking about have no causal interaction with us and therefore functionally don't exist).
...At least I think it is, I actually forget.
See I read in a book that Shannon Entropy is inviolable but I don't really remember the details or which exact conditions apply to that. I could resolve this uncertainty by going and looking up an authoritative resource which happens to state 'Shannon Entropy is inviolable by mathematical proof' and then rest assured there is definitely order and structure in the universe.
Or I could test if I really know it by trying to fill the details back in with you. Yeah lets do that instead.
Rather than start by introducing any weird symbols or theorems, it's probably better to walk ourselves into the core problem.
Imagine we're playing a game of chance, whose central mechanic requires recording the outcome of coin flips (we will assume a platonically ideal unbiased coin which comes up heads or tails exactly 50% of the time). The exact game we're playing doesn't really matter, but we do have to write these coinflips down. We might do that by writing a '1' if it comes up heads and a '0' if it comes up tails. In computer science we call these ones or zeros a binary digit, or bit. So we flip a coin eight times and get something like this:
01010111
And if our game is long we might end up with a lot of bits, it could get a little exhausting to write them out so it would be very convenient to have a shorthand notation for repetitive sequences that will occur every so often. Say something like:
0101 = ^01 1010 = ^10 1111 = ^11 0000 = ^00
If we are lucky enough to get a sequence like 11111111 we can repeat the caret twice for even better shorthand:
01010101 = ^^01 10101010 = ^^10 11111111 = ^^11 00000000 = ^^00
If we think about this for a moment the caret operator actually does the same thing in both cases, repeating the bitstring twice. ^00 becomes 0000, ^^00 becomes 0000000, ^^^00 could become 00000000000000 and in this way perhaps we can make the records of our game become arbitrarily small through multiplication?
The essence of Shannon's theory is to show that you cannot do this, that any attempt to do anything analogous to this will always fail, and that this is something you can expect to be true in any universe you might find yourself in. But before we can understand why it will help to elaborate on our game a bit. Imagine each round involves eight coin flips, so that it makes sense to separate them into rows like this
10101010 01010101 11110000 00001111 01101001 10010110
As it turns out these rows of eight bits are exactly how modern computers store their files, breaking up the bits into rows of eight called 'bytes'. The problem of finding a shorthand notation which can be reliably reversed to get the original binary digits is called compression coding, and it's a fundamental problem in computer science. Any time you've ever had to zip a file so it would fit into an email, or unzip a download using an obscure tool you've never heard of you interacted with compression coding. This thing with the notation where you replace longer symbols with shorter symbols is called a codebook, and it just so happens that codebooks become less and less effective at compressing noise (i.e. unbiased coinflips) the longer the noise gets.
To understand why lets return to a single coinflip, and why we're representing it with bits in the first place. By definition our unbiased coin returns heads or tails at equally likely rates, so half the time we will have to write down something that means heads and the other half we will write down something that means tails. There are two potential outcomes, so we can represent this with a symbol system that has two symbols to match the outcomes to, 1 and 0.
1 0
Now lets imagine we flip a coin twice. How likely are we to get any particular bitstring that can result from this? Accepting that our chance of getting a 1 or 0 is 50% either way, lets imagine we have already flipped the coin once and gotten an outcome, a 1 say. We now flip the coin again to get a second one or zero to go with our first, what is the likelihood of either outcome? It remains 50%, obviously. So in half of the cases where we get a heads we will get a second heads and in half the cases where we get a tails we will get a second tails. In other words adding a second bit doubled the number of outcomes that are possible. If before any outcome had a 50% chance of occurring, now it is 1/4 or 25%. We can visualize this for ourselves by writing out all the possible outcomes:
01 10 11 00
This principle of course extends to a bit length of three as well, which has eight possible outcomes:
000 001 011 010 100 101 110 111
A four bit string has 16 combinations, etc. An eight bit string or byte has 256 possible combinations. As a matter of empiricism nobody has written a program which makes files arbitrarily small (and our intuition would find such a thing absurd), but fusion power is generally accepted to be possible even though nobody has built a sustained fusion reactor (and our intuition finds the potential power output of such a machine absurd). So do we have a better way of knowing whether or not that's possible than social proof and intuition?
Yeah of course, you just learned most of it.
So lets imagine our 8 bit bytes again, neatly lined up row after row to describe a file which is pure random noise. Why can't a codebook compress it? Well for short files it probably can, because there will be irregularities in the frequencies of the different bytes just due to random generation not having had enough time to reach the long term average. You can flip a coin 8 times and not get 4 heads and 4 tails after all. No what we are specifically meant to observe is that in a sufficiently long file the opportunity for compression of random noise should trend toward zero. The reason why is fairly simple, compression codes take advantage of the fact that some symbols are more likely than others and as we've defined our file every byte is equally likely to occur. A byte of all zeros is as likely to be seen as a byte like 10110100, even if the latter is much harder for a person to remember. Therefore even if we use something clever like a Huffman Code, which finds frequent long bitstrings and assigns them to shorter bitstrings in its codebook we will not get any advantage as the file size increases because any given bitstring is inversely proportional in likelihood to its length. There's no regularity for the Huffman Code to latch onto, the most frequent strings are short strings and the most infrequent strings are long strings in exact lockstep to their length so that in a sufficiently long file the frequency of every byte or subset of a byte or combination of bytes occurs such that the code can only really replace short strings with equally short strings and long strings with equally long strings. There's also a short proof that any 'perfect' lossless compression algorithm has a larger set of input possibilities than output possibilities and therefore some inputs must be mapped to some outputs more than once and therefore the compression is not lossless.
But that's just codebooks, what if there's another way to compress noise, say we relax the requirement of being able to exactly reconstruct the original file and just focus on getting it mostly right, can we at least do that? Well no. Any lossy compression method is at least implicitly a form of prediction. Therefore the intuition for this one is even simpler: If you could you'd be able to predict the result of the next coin flip more than 50% of the time and beat the house at the game, which by definition can't happen if the coin is platonically ideal unbiased 50% random. We can test this for ourselves by say, training an LSTM neural net to predict the mapping of inputs to cryptographically random output labels but that experiment is just testing whether the cryptographic random number generator on your computer conforms to the definition of an unbiased coin or not so I'll decline to actually do it.
But does this have to be true in every universe, couldn't we find ourselves in a universe where we can somehow produce a codebook that maps the average random bitstring onto a shorter bitstring? And the answer is no because if you could it would break the definition of the random coin. If it was not the case that each entry in the codebook occurs on average in inverse proportion to its length you would no longer be observing the output of a noise generator, it would be some other kind of function that is just more compressible. Or to put this another way what we have just defined is the most disordered something can be, anything that was otherwise would actually be more ordered than this and therefore no longer conform to the definition of perfect chaos. In other words it's not so much that you would be breaking this 'universal law' as that you would no longer be in the kind of place where unbiased coinflips exist and this stops being useful for you to think about. To wit Chapman:
A perfectly objective meaning would compel all possible observers to agree. At minimum, you’d have to get buy-in from the eldrich abominations in the Andromeda Galaxy. But hey, they’re not so different, basically people like us, probably pretty reasonable once you get to know them. It’s imaginable that much more radically different entities are possible, and even if they don’t actually exist anywhere, to be properly objective you’d need to get them on board too. Elementary mathematical truths are candidates for perfect objectivity; not much else. (And can you be sure the eldritch abominations don’t have different and better ideas about mathematics than we do?)
I can in fact be pretty sure that the eldritch abominations don't have different and better ideas about mathematics than we do. Can I be really, truly certain that you can't compress the result of an unbiased coin flip into less than 1 bit in every possible reality? Admittedly no, it is always possible at least in principle that I'm stuck under some inductive bias that prevents me from seeing the cases where this could be a thing. However call it audacity, call it foolishness, but I have the courage and hubris to believe that two and two make four in every possible universe I have any business thinking about. Like is it possible that we are a simulation occurring in some kind of physics that is so alien that we could not even in principle describe it with any computable function in our universe given infinite time and resources? I guess. Is it likely?
Uh, no.
I mean for one thing when we want to make a simulation it has to in some sense be causally connected to our universe's physics somehow. A being instantiated in Conway's Game of Life could figure out plenty of things about the life world that apply to our universe too. In fact I'm fairly sure math just keeps working there. So we would have to posit that the parent universe is somehow ordered enough to simulate our universe but also simulating our universe through a method which has no causal relationship in its physics to the physics which it is simulating for us. This seems...I'm almost tempted to say it's actually impossible. I feel an invisible dragon or separate magisterium argument coming on. And even if it's not impossible, it seems extremely unlikely? Occam's Razor says we are most likely to be simulated by a universe which has physics in which our metaphysics are shared. It would be weird to just assume the most alien and incomprehensible parent universe possible. In fact usual simulation arguments go the opposite direction and claim that the simplest prior is that our universe is an exact copy of the parent universe, perhaps by future humans who are simulating us as ancestors etc.
If that all seems extremely esoteric and unrelated to anything else you will ever think about in your life it's important to realize the immense epistemological implications here. We know there is a certain size of representation which is necessary for an idea to exist, beneath which it is not possible for any form of knowledge to go. Perhaps the most important implication is that information theory gives us a rigorous set of heuristics about what can be realistically expected from formal systems. There is a Borgesian fable about an empire whose cartographers make a map of the country so detailed that its size covers the territory it is meant to represent.
Alfred Korzybski coined the phrase 'the map is not the territory' to describe the essential difficulties posed by abstraction. Korzybski was aware even then that no map of the world can be equivalent to its territory even in principle. It is simple enough to prove this by pointing out that a map of the world which completely and totally described the world would have to include the map itself, leading to an infinite regress. However Korzybski, who died in 1950, did not have access to the more rigorous framework provided by Shannon for thinking about this issue. Had he been writing some decades from 1950 he may very well have said that the map compresses the territory. Compression, with all the natural problems that compression implies. Sometimes it is possible to get a lossless encoding of an exact territory, but this is usually a property of territories and ontologies which are human artifacts to begin with. For example mathematical objects like functions can often be completely captured by terse notations. Consider the parity function, whose definition is so simple most children could be taught to compute it in a few minutes:
Given a bitstring of length n return 1 if and only if the bitstring has an odd number of ones, otherwise 0.
Parity(0110) = 0 Parity(1110) = 1 Parity(1111) = 0 Parity(1011) = 1 Parity(0000) = 0
This function, which might seem so simple it's odd that it has a name at all, has an interesting role in the history of AI. Research into neural nets was held back for some time by the observation that the perceptron, an early predecessor to artificial neural nets, is provably incapable of learning the parity function. Because the perceptron could only learn functions which are computable as a series of linear matrix operations, a proof that no series of linear operations could learn it convinced people that linear algebraic neural networks weren't viable. Later it was discovered that by introducing a 'hidden layer' between the input and output which was nonlinear you could in fact learn the parity function. A modern pytorch implementation of that looks like this:
<parity function network code>
I bring this up because it lets us continue from our exploration of noise to get an idea of what heuristics information theory gives us for learning structure.
The parity function is an instructive example because it is very simple to define but hard for artificial learners to understand, providing an opportunity to see what understanding is made of. If a neural net is substantially larger than the training data it is trying to learn from it will solve the problem through memorization. The resulting net winds up 'overfit' and incapable of providing the correct answers to new examples. Memorized knowledge is equivalent to a lookup or truth table, a simple mapping between known inputs and their outputs. When you train a neural network to compute the parity function you generally do so for a particular bit length, for example training the network to compute the parity function for four bits like 0101, 1010, 1111, etc. Holding the length fixed like this lets us conveniently compute the size of the truth table without having to write out every input and output (which becomes intractable as the bitstrings grow in length). For example, the parity function for length 16 bitstrings requires two bytes for each input and then one bit for each output. There is one input and one output for each possible 16 bit string, or 65536 possible values. That means the truth table fits into 136 kilobytes exactly.
Memorization provides a baseline worst-case for learning performance and the procedure to compute the parity function provides the best case. Using a procedure to define the best case compression for a concept is known as the Kolmogorov complexity measure. The shortest computer program which produces the truth table of the parity function as an output is going to be one which procedurally computes the function, not one that has its inputs and outputs stored in memory. Any reasonable articulation of this program is going to come out substantially smaller than 136 kilobytes, here's one in python that's 430 bytes:
def parity(input_): count = 0 for i in input_: if i == "1": count += 1 return 1 if count % 2 else 0 def gen_truth_table(n): t = [['0'],['1']] for i in range(n - 1): t_ = [] for j in t: t_ += [j + ['0']] t_ += [j + ['1']] t = t_ for i in t: bs = ''.join(i) print(bs + ": " + str(parity(bs))) gen_truth_table(16)
There's plenty of caveats here like using a high level programming language and longer than necessary variable names, but I doubt an assembly rendition would negate the fundamental point. It's not important what the exact theoretical minimum representation is, just that it is not very large. As the bitlength n increases the truth table becomes exponentially larger but the python program remains about the same size (a byte would need to be added for each order of magnitude we increase n by). It is also not particularly important how efficient this program is in either memory or CPU cycles as we are focusing on size of conceptual representation not performance. Between 136kb and somewhere in the realm of 430 bytes lies the economy we can expect from our neural net, which at its most efficient can hope to approximate the economy of algorithmic representation. Because the parity function is plucked from the platonic realm it is reasonable to expect certain neural architectures to learn a perfect representation of it. However even if a territory can't be boiled down into a perfect algorithm it is still generally possible to learn some approximation of it, if this weren't the case it is doubtful our minds would exist in anything like the way they do.
Now we can begin to understand what's happening when the Greeks replace that Babylonian library with math. The Babylonian library is a truth table of architecture, a memorized series of inputs and their binary output (did the structure fall down? y/n). As the Greeks begin learning more of the physical principles that go into engineering the amount of writing you need to represent the 'function' of architecture decreases. Algorithmic and principled understanding replaces rote memorization, bringing the economy of expression down and down and down until building a structure that stays standing goes from an art no one could hope to truly master in their lifetime to the content of an undergraduate degree. If you deny the awesome power of this by pretending it is somehow 'equal' to taxonomy or the 'wisdom of the ancestors' (which ancestors? the Greeks are our ancestors buddy) then again you are not 'metarational', you are a romantic.
So when Cartwright asks how unified physics is, from the perspective of formal systems, and it's either unified or it isn't, this is just entirely the wrong way to ask the question. The question she should be asking is: How compressible is our universe's physics? And Cartwright's implicit answer in her later essays is "pretty darn compressible", she thinks that physics gets its apparent unity by finding rules which humans happen to be able to work with using their weak cognitive resources, and there is just no way that is not an implicit admission of relatively low kolmogorov complexity. If that is possible at all then there must be extreme regularity to the patterns observed in the physical universe, even if we bite a bullet by accepting the claim as given that there are artifacts which don't represent the underlying territory when you look at it up close.
Often people will ask when AI becomes 'conscious' as though 'consciousness' is the thing that gave humans their power to invent and act strategically. The right question to be asking is when will we build a goal directed prediction machine which can do high fidelity lossy compression of our universe's physics. When you phrase it as 'consciousness' the problem seems mysterious and impossible, but once you put it like that this starts to seem much more tractable. You may ask what about qualia, what about experience, and the answer is that as far as AI goes these things don't matter. The goal of AI is to make a thinking machine, not a feeling machine. If it is materially possible to build a machine without those things it will be.
Buddhism, Heidegger, "Traumatized By Objectivity", etc
Eliezer Yudkowsky's rationality is a very specific set of mutually reinforcing philosophical positions. It's hypermaterialism, so if we're going to get phenomenological about what 'postrationality' as in "was a LessWrong rationalist and no longer am" is, it's the process of coming to disagree with Yudkowsky's hypermaterialism such that one falls out of the LessWrong overton window. There's a lot of ways for that to happen, as I've written before:
There are two ways you exit the 'rationalist community', one is by deciding it's too much of the thing and dropping out. The other is by deciding it's not enough of the thing and becoming too much for everyone else. https://t.co/dJcroywQ71
— John David Pressman (@jd_pressman) January 26, 2021
I'm personally in the latter category, postrats are in the former category. I've already discussed disillusionment with rationality as a strategy (illegibility, anti-induction, et al), and I've discussed the extremely common sophist argument that we don't have any rigorous rationale for expecting the universe to be consistent. But I think that the sophistry pretty much always comes later, it's an excuse to justify a change in belief that has already happened, rather than something which changes minds on its own. Usually what people do once they've been disillusioned is move on to Buddhism or (less common) Catholicism. The Catholic types are typically known to history as the Neoreactionaries, who I'll limit my commentary on to noting that the life of Ernst Junger is instructive and otherwise pretend they don't exist. The Buddhists on the other hand I have a lot to say about, but first it's worth asking: Why Buddhism?
To explain the draw of Buddhism for disillusioned hypermaterialists, it pays to recall that LessWrong rationality is a distant descendant of General Semantics. General Semantics was a movement which attempted to accelerate the improvement of societal institutions by teaching people to think better. 'Think better' in General Semantics meant teaching people to distinguish between layers of abstraction in their problem solving. For example noticing that there multiple senses to words, and often we will confuse ourselves by using the same word to refer to multiple things in the same sentence. General Semantics played a large role in the 1940's California sci-fi zeitgeist, where it joined a cottage industry of ideas like Scientology which promised superhuman powers through reason:
What all these threads shared was the idea of the system as a particular kind of mail order DIY transcendence: a promise that with the home study and practice of rational experience you, too, can become superhuman. Van Vogt’s scene shares a kind of Sears-&-Roebuck-Catalog-of-the-mind vibe with productions like the Whole Earth Catalog – a way to join the evolutionary vanguard from the kitchen counter. But where Whole Earth presented guides to street theater, composting, home birth, and yurt construction, the van Vogt fantasy was always internal: that you could, by systematic mental training, become smarter, more rational, clearer, and through that lay a life transformed without ever having to leave your desk. This was a very seductive promise for people already convinced of their superior intelligence, some of whom began to found their own cooperative houses – to put these plans into practice …
In this light LessWrong and Buddhism are both similar sorts of things, they're what Hinduism would call knowledge yoga. You contemplate things very hard and gain fundamental insight into the nature of reality that transforms your life.
For many 'aspiring rationalists' the transition into Buddhism and psychotherapy was gradual and came out of trying to solve problems they were encountering with their rationality self improvement. I think in retrospect one of the hardest things to understand for anyone trying to parse the 2015-2020 'era' of LessWrong rationality (both online and in person) is the extent to which a lot of the weirdness and psychotic meltdowns come naturally out of an environment where people are really trying to see past their own blindspots and limitations of perspective.
LessWrongers to an unusual extent were people who would engage each other at length in an attempt to root out flawed behaviors and shoddy thinking. Aumann's Agreement Theorem drove me and many other young people involved in the movement to exhaustive hours trying to get to the bottom of differences in perspective. I'm somewhat convinced a lot of the early Internet's magic was its ability to reliably produce this dynamic in strangers. In the realm of personal development this often meant that when you did something like drill down into the most important thing you see that the other person isn't seeing, there was an entire constellation of problems keeping them in an equilibrium of not seeing what might seem intensely obvious from the outside.
Overcoming an equilibrium of mutually reinforcing mistakes, traumas, and biases requires a lot of creativity and effort, even just from a basic argument perspective. It's not enough to say "it's really obvious from the outside that your views on this subject are coming from an objectively irrational belief you were taught in childhood", you have to find a clever way to express this to the person in their own subjective mental language so that it doesn't get filtered out or fail to resonate. Which is to say that beyond a certain point improving your epistemology with others runs into the flaws in someone's implementation of cognition more than their object level beliefs. This led to, among other things, adoption of mental illness concepts and Tumblr-style insight seeking becoming more important to people than raw statistical methods or biases. 'Bias' went from being an objective, human-level concept about everyone's shared hardware to a subjective attempt to self-help ones personal problems.
And that's all fine.
Sorta.
Don't get me wrong 'secular spirituality' strikes me as very cringe, Bay Area group housing and its consequences have been a disaster for the human race, and I've seen more than my fair share of friends descend into psychosis and dysfunction after doing too many psychedelics and half understood meditation practices. In some respects I'm properly pissed, and the lifelong grudge I've developed over it is the basic fuel for writing this post. But in other respects it would be totally unreasonable for me to expect anything else. After all I just described the logic behind the transition from 'objective' statistical methods to subjective therapeutic and 'spiritual' ones. With the benefit of hindsight it's difficult to imagine the movement evolving in a different direction because by the time it resorted to this stuff it had already basically failed. It's not so much that I don't have critiques of these things, as that I think in 2022 they've had ample time to embarrass themselves and advocating 'rationality' as a replacement would be boneheaded.
{ ... stuff happens here don't know what }
Qiaochu Yuan (hereafter QC) has a semiviral thread in which they explicitly call out LessWrong/MIRI/CFAR/'the rationalists' as an unhealthy cult. This thread gets brought up a lot on Twitter whenever somebody opines on what postrationality 'is', probably because of this tweet:
the esoteric meaning of "postrationalist" is a person who is recovering from the totalizing memes in the rationalist ecosystem. it is a trip. especially because you can't just refute them by saying AGI is impossible
— QC (@QiaochuYuan) January 25, 2021
I want to be gentle here for a few reasons. QC seems like a nice person, and they're clearly very hurt by what they experienced. At the same time it's important to have some context about the speaker, QC by their own admission did a Ph.D program where they delved deep into the purest theory wank mathematics has to offer, and it took them into a deep depression:
I had a kind of like quarter life crisis in grad school as I was thinking about this, I was just like: why am I doing this? Like the specific subject I was working on was like so abstract that I was pretty sure there were less than four people on the entire planet who would be prepared to understand the thing I was trying to do. If I did it! I didn't do it, I made very little progress on that project, but even if I had done it like just so — the audience, the potential audience for that paper was so tiny. It was like: My advisor, one guy at Harvard, maybe another guy at Harvard, and like no one else was really gonna be prepared to understand it. And it just seemed really pointless.
Konspiratsiya
JC DENTON In other words, yes. You want me to blow the facility up -- why? It's just a hole in the ground. TONG Decades ago, the UN made Area 51 the central hub for all electronic communications. The Aquinas Protocol, originally for surveillance, has given Page unlimited abilities to censor and control all forms of media. JC DENTON If we destroy the Aquinas Hub, we'll take down the global network. TONG Exactly. They dug their own grave, JC. We're going to eliminate global communications altogether. As long as technology has a global reach, someone will have the world in the palm of his hand. If not Bob Page, then Everett, Dowd...
— Deus Ex (2000)
I want to level with you for a minute.
This "illegibility" thing? It's not going to protect you.
When you're browsing Twitter with your pseud and a cryptic bio it's very easy to feel like just another face in a big crowd. But the truth is that is not how a state, let alone an authoritarian regime, heck not even how Twitter itself looks at things. When Twitter wants to figure out whose account to terminate in their latest banwave, they do not have an army of spooks crawling over the network looking for suspicious individuals. They have a database of accounts: who is followed by and what those people look at. And what they do is they run a handful of simple matrix operations to pick out who it would do the most damage to ban or suspend, and they mete out their punishment. You have already made yourself and your social relationships and your interests legible just by interacting with the site and reading the stuff that's compelling to you. That is the point of social media.
The truth is, you think you're not getting banned because you're one step ahead of the corrupt authorities when in actuality the corrupt authorities are one step ahead of you. They've decomposed Twitter into its component cliques and deliberately pick them off slowly so that you don't move your communications somewhere else. They have a small army of stats wizards who have figured out the exact right way to slit your throat so your 'community' bleeds out without anyone noticing something is amiss. And you think they're restricted to just banwaves? Just a few matrix operations on your liked tweets yields a strong prediction of content you won't like, progressively sprinkle more and more of that into your timeline and you might think the site has just 'lost its magic', they'll get you to leave entirely on your own initiative and think you're independent for doing so.
You're also not the first people to react this way to enchroaching totalitarianism. Do you have any idea how many Jews tried slipping into the shadows before Hitler's genocide? They thought they knew what was up, centuries of persecution give you a good sense for when a pogrom is coming. But unlike so many pogroms before, this time the regime made a list and checked it twice to make sure they exterminated every Jew in Germany they could lay their hands on. But did you ever ask yourself: How'd they get that list anyway? It was simple really. Expensive, but simple. All they had to do was seize records of births, Synagogue marriages, genealogy, and other data so it could be fed into their new card sorting machines leased with staff from IBM. Once they had a fairly good idea of where the Jews were, a census was taken to fill in the blanks until they had a nearly complete hit list of every Jew in the country. Plenty of people had no idea they were even Jewish when the Nazis arrived to take them away.
If the woke police switched the flip and decided it was time to dispose of the kulaks, the truth is you and all your friends would be dead.
Humanism probably died at some point during the 20th century. It arguably breathed its last gasps in Lenin. You know who Lenin is right? On his way to becoming dictator of Russia he came up with this whole theory of revolution, at the foundation of which was konspiratsiya. Konspiratsiya is a russian word that translates into something like 'opsec' in English. Lenin's theory of revolution was premised on opsec. The logic being that if the repressive state police were going to follow you all over Europe and monitor your mail and infiltrate your organization, there was simply no way you'd ever take down the Tsar without professional spycraft. Therefore a revolution of friends was out of the question, you needed professional revolutionaries whose entire being went into the cause.
Lenin was a NEET who traveled across Europe in secret, working out of hotel rooms and apartments to write blog posts (then called papers) and manage his sprawling criminal insurrectionary empire. His agents smuggled newspapers, guns, and other contraband into Russia. They organized into local terrorist cells and appropriated resources from the state in strings of bank robberies and other heists. Meanwhile Lenin himself survived on funds sent by his extended family to keep him going. He walked on tiptoes while he paced his cramped workspace so as not to disturb his own thoughts: like any good autist he had odd hypersensitivies.
And the entire time he really was the #1 target of the Russian state police, the Okhrana, who were monitoring mail and counterculture all across Europe in an attempt to capture him or disrupt his operations. But even when sensors are cheap attention is expensive, and it was just as often that Lenin could get some of the heat off himself by lying low for a while. Helen Rappaport writes in her incredible intimate biography of Lenin:
It was hard for the Swiss police to keep tabs on them all, for Switzerland in 1917 was swarming with artists, revolutionaries, bohemians, and spies, from professionals based at the embassies of the belligerent nations to waiters, serving maids, and domestic staff in the hotels. They were bribed to watch, look, and listen. Everything was reported, everything was supervised as agents circulated their daily reports, telephones were tapped, and "wastepaper baskets and blotting pad correspondence was sedulously reconstructed." No one, however, was taking much notice of Lenin at the time; his deliberately inconspicuous lifestyle near a sausage maker on Spiegelgasse was part of his tried and trusted conspiratorial method of blending into the background. The Austrian playwright Stefan Zweig, who had fled to Zurich because of his pacifist sympathies and saw Lenin from time to time at the Cafe Odeon, later recalled wondering how this obstinate little man ever became so important.
That 'inconspicuous lifestyle' relied on a mountain of work and supporting infrastructure. From roommates that would spend hours encoding and decoding secret messages to networks of safehouses ferrying mail and political exiles, the European underground involved an impressive level of effort. Religious levels of opsec and personal penury were practiced by the Bolshevik fanatics on their way to revolution. Reading about it today one can't help but notice that it would be nearly impossible to convince people to put the same amount of effort into anything, let alone an insurrectionary force in the 21st century. Much of how Lenin came out on top was his strategy of organized crime allowing him to pay Bolshevik organizers a real salary, while other socialist factions struggled to empower their followers with money.
And all of that represented the state of play in the late 19th, early 20th century. Revolutions, insurrections, underground countercultures, that stuff was largely played out by the end of WW2. 'Revolution' became a standard tactic in US and Soviet imperialism during the Cold War, stripped of any emancipatory function separate from promising a subset of key supporters a greater share of the regime's treasure. The ultimate outcome of an advanced technological civilization, the very phrase 'advanced technological civilization' meaning more and more precise control over marginal matter to the nth decimal point, implies a world where problems stop occurring at the human scale in the natural human modalities. Your sense of sight tells you that when you don a mask on Twitter you're hiding yourself, but the invisible web of relationships you've cultivated mean you glow like a Christmas Tree to its managers. As Nick Land puts it:
Level-1 or world space is an anthropomorphically scaled, predominantly vision-configured, massively multi-slotted reality system that is obsolescing very rapidly.
Garbage time is running out.
Can what is playing you make it to level-2?
In such an environment it's not clear that 'becoming embodied' is even desirable. Increasing your reliance on the natural human sensory modalities is a liability when so much more effort has been put into subverting them than could be provided by the ancestral environment. Roussaeu's return to nature is ultimately a return to the state of being understandable as a matter of natural philosophy, it is deliberately putting yourself in a place to be optimally exploited. Natural behavior is predictable behavior, easily controlled, with many parasitic strategies ready to take advantage. If strong people display 'natural' behavior, it is a luxury they can indulge in on the back of their extreme advantage. You will no more become strong by reverting to nature than a poor man will become rich by buying an expensive car. It is a complete confusion of cause and effect.
