Loper OS

I find Clojure revolting.

It is the most explicit to date abandonment of the age-old Lispers’ Dream, “Lisp All The Way Down.” Clojure is the antithesis of the Lisp Machine. Instead of a crystalline pyramid of comprehensible mutually-interlocking concepts, behind every Clojure primitive there lurks Black Magic. The Clojure user who is missing some routine or other will run crying to Java for help, rather than implementing the feature himself correctly - that is, as a natural part of the entire language, built on the same concepts. Clojure pisses on everything I’ve ever loved about Lisp. It rejects even what little consistency and intellectual rigor there is to be found in an abomination like Common Lisp.

Clojure is the False Lisp, which Reeketh of the Cube Farm. A Lisp unworthy of the name; one which encourages users to pepper their code with calls to opaque routines having no underlying Lispiness. A Lisp which barfs Java stack traces. It promotes - no, mandates - the use of undigestable foreign matter in Lisp code: primitives on which you cannot pop the hood to reveal intelligible innards.

I don’t care if everybody really is more productive in Clojure than in Common Lisp. The latter is not my standard of comparison (Symbolics Genera, the state of the art, is. Or, if you want something that is distinctly inferior but is currently available on the market: Mathematica.) Clojure is the gutted and zombie-reanimated corpse of a dream.

The cult of Good Enough which seems to pervade all of modern computing has finally chewed its way through to the Lisp community, with Clojure as the result. I am tired of this abomination being hailed as the future of Lisp. Aficionados of real Lisps, such as myself, will keep hoping, dreaming, and working on systems which do not betray the original virtues of the language.

Symbolics Genera remains the gold standard of programming systems. Though largely forgotten, it can never be un-created. My standard of comparison for any technology will always be everything previously achieved by mankind, rather than what is available on the market today.

Lamport: “The Future of Computing: Logic or Biology?”

“We understand automobiles. There are no homeopathic automobile repair shops, that try to repair your car by putting infinitesimal dilutions of rust in the gas tank. There are no automotive faith healers, who lay their hands on the hood and pray. People reserve such superstitions for things that they don’t understand very well, such as the human body… … Superstitions arise because we don’t understand. If this trend continues, we may soon see homeopathic computer repair. Perhaps it will consist of adding a virus to a program, deleting the virus, and then running the program. You may soon be able to take your laptop to a faith healer, who will lay his hands on the keyboard and pray for the recovery of the operating system. Is this the future of computing? Or can we instead build on the idea that a computer program is a mathematical object that can be understood through logic?”

Weinreb: “Why did M.I.T. Switch from Scheme to Python?”

“Nowadays, a real engineer is given a big software library, with a 300-page manual that’s full of errors.  He’s also given a robot, whose exact behavior is extremely hard to characterize (what happens when a wheel slips?). The engineer must learn to perform scientific experiments to find out how the software and hardware actually work, at least enough to accomplish the job at hand.  Gerry pointed out that we may not like it this way (”because we’re old fogies”), but that’s the way it is, and M.I.T. has to take that into account.”

Programmers have managed to create a universe where “science doesn’t work.” Superstition will thrive there, because the human mind demands pattern; it insists on at least the illusion of understanding, of predictability. Learning where the permanent bugs and workarounds are inside a phonebook-length API teaches you nothing. It is anti-knowledge. Where your mind could have instead held something lastingly useful or truly beautiful, there is now garbage. Most of what passes for learning in today’s computing field is ultimately worth about as much as a primitive shaman’s repertoire of magical rituals.

This is a repost of my  YC news comment on the subject.  Lame reposting is not a capital offense (yet…)

My first encyclopedia when I was young was a CD-ROM edition of Britannica. It had a superficially deep article on almost everything, and made for endless hours of joyful exploration. Then one day I found a complete 1958 paper edition of the Britannica in a dumpster - in perfect condition, no dirt or damage. It was an entirely different world. The degree to which it surpassed the digital edition defies description. The articles on various machines, chemical reactions, fortifications, etc. seemed detailed enough to replicate all of these wonders with my own hands, had I the time or resources. The coverage of most historical events seemed intellectually top-notch and well-researched.

That was when I first understood that the transition to digital media is not an unmitigated good, and that much is being lost - without many people necessarily noticing. I still have the 1958 Britannica, and I intend to keep it for the rest of my days. Some of the articles inside (in particular, on mathematical concepts) are unmatched in their quality by Wikipedia or even Wolfram MathWorld.

The image on the left is the original 1890s NYC grid of power and telegraph cables, built by a hodgepodge of competing wire-running firms. A blizzard blew all of it down, causing chaos. After this, the city decreed that all cables are to be buried and passed regulations governing how it is to be done. The picture on the right is of the city after this had taken place.

The construction of multiple electric grids in a city is an almost tragicomical duplication of effort. And yet, we see a very similar situation in today’s computer software, where a single machine may have a dozen simultaneously operating memory allocators / garbage collectors, graphical subsystems, web content renderers, and many other essential services which in a sanely designed system would exist in a “one true” incarnation, provided by the OS itself.

Yossi Kreinin throws down the gauntlet to all those who believe that a CPU ought to be designed specifically around the needs of high-level languages:

Do you think high-level languages would run fast if the stock hardware weren’t “brain-damaged”/”built to run C”/”a von Neumann machine (instead of some other wonderful thing)”? You do think so? I have a challenge for you. I bet you’ll be interested….
My challenge is this. If you think that you know how hardware and/or compilers should be designed to support HLLs, why don’t you actually tell us about it, instead of briefly mentioning it?

In his excellent follow-up to the challenge, he argues - fairly convincingly in my opinion - that a move toward “high-level” CPUs would come at substantial cost, and lead to no miraculous speed-up of dynamically-typed code. However, I still believe that ultimately, today’s architectures ought to be consigned to the junk heap in favor of a resurrected multi-GHz Lisp Machine descendant. The best argument for this is hinted at in another one of Kreinin’s posts:

How many problems did you have with hardware compared to OS compared to end-user apps? According to most evidence I got, JavaScript does whatever the hell it wants at each browser. Hardware is not like that. CPUs from the same breed will run the user-level instructions identically or get off the market. Memory-mapped devices following a hardware protocol for talking to the bus will actually follow it, damn it, or get off the market. Low-level things are likely to work correctly since there’s tremendous pressure for them to do so. Because otherwise, all the higher-level stuff will collapse, and everybody will go “AAAAAAAAAA!!”

Hardware really does tend to be the product of a more… adult engineering culture than software. I don’t believe I’ve ever encountered a piece of computer hardware which was a steaming turd of sheer dysfunction (non-deterministic behavior, illogical/undocumented operation, simple defectiveness) in the same manner the average piece of software of any substantial complexity almost invariably is. On further contemplation, I can think of a handful of products which might qualify - yet in each case the fault lay in firmware - once again the foul excreta of software engineers.

Bridges are expected to stand up, and on the “first try,” even! Planes are expected to stay aloft. And yet programmers seem to be content with forever competing in the engineering version of the Special Olympics, where different, “special” standards apply and products are not expected to actually do what they say on the box - at any rate, the idea of offering a legal warranty of proper function (or even of not causing utter disaster, in the manner customary in every other industry) for a software product is seen as preposterous.

I see this as a convincing argument for “silicon big government.” Move garbage collection, type checking, persistence of storage, and anything else which is unambiguously necessary in a modern computing system into hardware - the way graphics processing has been moved. Forget any hypothetical efficiency boost: I favor such a change for no other reason than the fact that cementing such basic mechanisms in silicon would force people to get it right. On the first try - the way real engineers are expected to. Yes, get it right - or pay dearly.

The author of LoseTheOS comments on the lost treasure of simplicity in computing:

“When I was a kid, you could put a line on the screen in two lines of BASIC code.  I had a book which literally told what everything… in my computer did. … Average people typed-in programs from magazines and, typically, knew enough to modify them.  The owner’s manual to the computer told how to program it. …My critics making their own operating systems… what distinguishes yours — why do you bother?”

As far as I can tell, he was responding to the horde of raging ignoramuses which descended on him after his site was linked from Reddit and YC News. They demanded to know why he produced what he did instead of disgorging yet another interchangeable Unix clone.

What follows is an essay which I wrote in a fit of rage at the state of current computer technology roughly a year ago. It is entirely unpolished, unedited, and probably contains quite a few words and concepts meaningful only to myself. I originally had no intention of ever making it public, but after reading this post I could no longer resist the urge to do so.

The printed word is an excellent presentation medium but a rather shoddy expression medium. Linearizing a thought necessarily destroys structural information. Unfortunately today’s technology does not allow us to share our thoughts with others save by running them through the meatgrinder of linearization and textual representation. This, however, need not always remain the case.

IA, or Intelligence Amplification [insert graphic of Dilbertesque cube drone balancing Einstein-esque figure complete with chalkboard on a “justice scale”] is AI’s shriveled mutant sibling, hidden away whenever there is the slightest chance of polite company appearing for the evening. The usual argument used to dismiss the contemporary computer as a vehicle of IA is bandwidth limitations. We flesh sacks are, supposedly, incapable of communicating our flesh bag thoughts quickly or unambiguously enough to satisfy a machine in real time, or at least to the degree required for a functional equivalent to a bona fide brain upgrade.

We struggle with keyboards and mice, both of which solve the wrong problem. The latter being “how do we encode visual representations of neolithic speech patterns?” The real problem is: how do we arrive at the canonical representation of a concept, in all of its stages of life - from conception, with all the infinite vagueness this entails, to a ball of conceptual clay, infinitely malleable, to a solidified whole suitable for use as a building block for further thought.

There is no bandwidth problem inherent to the human body and brain. All that must be done is to simply adapt the same sort of Huffman-like coding that forms the basis of all language to the communication between man and machine - in the most general possible form.

The key concepts involved in designing the ultimate and most general form of information manipulator are:
1) the “magic glove box” and 2) the Huffmanator

The glove box, for the lack of a better name, aims to utilize the two keenest senses possessed by mankind: vision and motor/tactile (hands, fingers - motion, touch. temperature?) Your arms and hands fit through two openings in what appears to be an ordinary glove box of the type found in biological laboratories. Unlike the latter, your first sensation is as though the box is full of warm water. You can rest your arms and hands without discomfort. Your seat is configured in such a way that you can sustain such an idle position indefinitely, again without discomfort. You could even fall asleep if you wish.

The first illusion is the transparency of the glove box. Although you can see your arms and hands, you are not actually looking through a glass sheet. You are, in fact, looking at a high-resolution color display. There is also no water inside of the box; nor is it filled with any other fluid. Your appendages are, in fact, resting on thin air, supported by arm-length gloves assembled from (ferrofluid? muscle wire?) microcells that can shift in position, stiffness, angle, temperature. Your sense of touch - humanity’s strongest ally in its toolmaking shenanigans throughout history - is your guide. As are your eyes. But the normal physical limitations no longer apply. The box switches into demo mode and displays M. C. Escher’s famous paradoxical staircase. You feel the staircase with your fingers, and it is actually there! You are holding, touching a - physically impossible object. And all of the sensations are correct, exactly as though our universe actually allowed the staircase to exist as drawn - no, more than as drawn; as implied by the drawing, in the same fashion as the Klein bottle is merely implied by the meager picture we are able to draw in three-space.

With the glove box you are able to touch anything you can see, with no limits imposed by the bothersome geometry of our universe.  Now what remains is to gain the ability to see - and record, and manipulate - anything you can think, anything you can imagine. There must be no thought that you are not able - within seconds at most - to place inside the box, at your slightest whim. If you picture a Platonic sphere, it will appear. If you want to deform it, or turn it into (what you remember of) a hemoglobin molecule, you can also do that. How could such a thing be possible? It is very easy. All one needs to do is to exploit the basic idea behind all language: compression. We already have a direction to travel in: the world’s sign languages will show us the way.

Sign languages replace the spoken word with hand gestures. The degrees of motile freedom of each hand/arm/finger joint multiplied with the others combined with real-time visual and tactile feedback from the glove box gives you the hands and eyes of a god. Now all that remains is to create a self-consistent universe for the newly minted god to play with. This is our next objective.

The first task is to descibe the outermost magisterium of thought. The concept of “concept”. Gather your fingers in the box, as though you saw a planar surface - an angled glasss table? inside. Angle your fingers, bunched together, as though you intended to place them on the glass. You see the glass. There is no telepathy - the machine simpy matched the anticipatory motions of your fingers with those recorded from humans who are reaching out to touch just such a glass table - and an idealized representation of it has been drawn before your fingers hit the virtual surface. You feel the glass, smooth and cool to the touch - the 0,5mm^2 peltier pixels on the inside of the gloves are doing a bang-up job. Your next act is to dig your fingertips into the middle of the glass, as though you saw a very small hole there and wanted to enlarge it. The glass spreads like modeling clay - except, the entire infinite plane does so at once. The physical limitations of real glass (or any other material) hold no power here. As you tear spacetime open, you contemplate what you should fill the void with. You curl your left hand into the form of a ball; your right hand curls into the shape it would be in if it were placing such a ball on top of an infinite sea of melted wax. You can feel the ball of your left hand in your right. You let go of the phantom ball with your right hand, and it drops - a new one taking its place a second later. You arrange the atoms in 3-space. The glove box supports your arms and hands in any position you may want to leave them in - gravity is irrelevant. A single pedal under your feet (think Singer sewing machine) adjusts your ability to see through solids. With the aid of these tools, you create simple subunits (as we just saw, a chem textbook style “atom”), place them in arbitrary locations in 3-space. Now you want to save them for future reference and use as building blocks. This is where the Huffanator comes in. In much the same manner as you created the shapes, you can also create concepts.

There is no need to type words, or represent ideas in textual form. A sphere appears whenever you want one, and so can an 1895-vintage dynamo - the only difference is a dozen or so conceptual links which you compose as easily as you composed the hemoglobin from the “rock paper scissors” shapes your hands started with. Your hands, with the feedback from the glove box, are all-powerful. You can lay them on any object - any piece of information, in fact - that you can imagine. Your fingertips slide over the prime numbers. The prickly stubble of the Riemann zeta function bristles against your skin. A flick of your wrist sends the entire universe, crumpled, like a fast food wrapper, tumbling into the void (actually, into permanent memory, but that’s beside the point.) As for the compression, that part is obvious. The machine learns the meaning of your most minute motions through frequency analysis - The Gold Bug - and attaches meanings to them. You provide it with nonstop confirmation or refutation of these meanings as you work. And the machine becomes you, and you it.

————————————————-

How to build the Glove Box
—————————-

One possibility is a fractal structure with the Stewart platform as the basic unit: larger ones for macro-scale movements, and vanishingly small ones for submillimeter tactile feedback.

… to be continued.

“The Jolitzes believed that from its origins as a series of quick, if elegant, hacks, Unix had hardened into a series of unquestioned rituals for getting things done. Many of these rituals were fossils — work-arounds for hardware that no longer existed. “It’s amazing how much we were still tied to the past,” Lynne says. “The physical machines had been hauled away, but elements of the operating systems are still being implemented in the same way.” She believes today’s Linux and BSD developers are carrying on that unhelpful tradition.” “It’s like they’re doing incantations,” she says. “They’re repeating what they’ve been taught, and they don’t know what it means.”

What will it take to re-bury the rampaging Unix zombie horde?

The aim of Loper is to build a sane computing environment on top of the ubiquitous yet nauseatingly flawed X86-64 architecture.  I believe that it is possible to abstract away its most damning shortcomings, such as the lack of direct hardware support for capabilities, orthogonal persistence, type-checking, and garbage collection.  However, wouldn’t it be nice if we were not confined to a machine composed of a quarter-century of layered anachronisms and hacks, and thus had no need of fancy workarounds?

 I have entertained such notions before. However, this discussion has prompted me to attempt the creation of a group specifically to collect and discuss related ideas. Let’s demolish the shanty town and build a palace. What would a computing system rid of all the brokenness with which we have grown comfortable be like? Assume nothing of the old ways must remain save the use of integrated circuitry and familiar logic design constraints. I now invite my handful of readers to join me at:

Sane Computing

Visit this group

Loper should not be considered truly dead until I myself am dead and buried. Currently available computing systems are brain-damaged in such wholesale, unmitigable ways that I am driven back to the project again and again, despite the oceanic size and nearly certain futility of the task. Working full time and being back in school (in a field unrelated to computing) have slowed my Loper work to a crawl. Yet I try to use my scarce free moments to think - and most of the work which must be done here will consist of thinking, rather than coding.  This is because my ultimate aim is to drain the complexity swamp in which programmers have drowned. At any rate, the project should be considered a failure if the alpha version exceeds, say, 20,000 lines of code.

   Important conceptual progress has been made by others. In particular, there now exists a working experimental operating system, “LoseTheOS“, which, among other virtues,

• was written from scratch
• tailored to the AMD-64 architecture
• does away with most aspects of the compile/pray/debug cycle
• does away with the programs-as-ascii-globs dogma
• is in the public domain.

I fully intend to build on this progress and learn whatever lessons there are to be learned from this masterpiece.

  Mencius Moldbug of Unqualified Reservations once remarked in his criticism of Haskell and other products of modern academia that most programming “on the ground” consists of juggling database queries. If, as I suspect, this claim is even partly true, this fact alone proves that most programming is entirely unnecessary and that the livelihood of most programmers is quite like that of the glassmaker in the Parable of the Broken Window.

The bootloader is complete, as previously announced. Where to now?

• Initialization of 64-bit memory management data structures
• Basic display driver for QEMU, Bochs, and VMWare virtual SVGA cards
• PIO disk access driver
• And, of course, the Loper Lisp JIT compiler. “First, steal a chicken…”

Expect more in the near future.