’Twas the night before Dub Dub, when all through the house, Not an iPad was stirring, not even with a mouse… – Stephen Hackett, 512 Pixels (link mine)
This is a continuation of the Harbingers series – an exploration of Apple’s advancements in Human-Computer Interaction and their influence on the computing industry. I’ve been working on Harbingers for over a year now, and I told myself I’d finish it before Apple announced their headset. Whoops! It stands alone, but builds on the ideas from Part 1.
A Pre-WWDC Harbingers Interlude
or: how i stopped worrying and learned to love the name
The conceit of this series is that human-computer interaction (HCI) is the most interesting part of computing innovation. Its assertion is that the metaphors we use to define user interfaces have profound effects on the world. Its vehicle for this exploration is Apple’s contributions to HCI. And I’m publishing this the day before WWDC 2023 because I believe we are on the literal eve of one of these paradigm shifts – one that is 39 years in the making.
In Part 1 of the series, I traced the contours of the Finder’s development, which drew inspiration from three major sources:
- Most famously, Xerox PARC’s GUI. The story of Steve’s visit to PARC is a bona fide legend. There, Jobs and the Lisa team at Apple saw the graphical user interface for the first time and the team fell in love.
- The MIT Speech interface Group’s Dataland concept. In more folklore about the sibling Mac and Lisa projects, Dataland often comes up as a philosophical influence on on the Finder’s spatial data management technique. This is a classic John Siracusa hobby horse, and has become one of mine as well. It’s easily overlooked how fundamental a shift it was to find data by going to where it is instead of by finding it by its name, because this interaction method universally became the foundation of consumer data management after the Lisa. And, finally –
- IBM’s Pictureworld concept. I believe this is the most often overlooked of the three major influences on the Finder, but I think that it is the most interesting. At the time, the Lisa project’s main goal was to make computing accessible to office workers who likely had no prior computing experience – a herculean task. Pictureworld offered a remedy for this problem, by leveraging a GUI to create an on-screen diorama of an office using icons. Whereas Dataland allowed a user to literally place documents in an infinite plane, with no “icon” representation, Pictureworld offered a more pragmatic approach for the 20th century: use icons to represent documents and applications, and layer a cohesive “office” theme on top of these icons (trash cans, inboxes, filing cabinets, folders, desk tops, etc.) to tie everything together. Pictureworld is what invented the desktop metaphor itself.
None of these influences formed a complete product, which is why it took Apple to synthesize all three before any one of the ideas took off.
In general, a metaphor stands or falls based off of its explanatory power. In designing Lisa, Apple knew it was important to “find a central metaphor that’s so good that everything [aligned] to it. [After that,] design meetings [were] no longer necessary, [the device designed] itself”. “The metaphor,” they thought, “should be crisp and fun.” […] The GUI, spatial data management, and the desktop metaphor were three revolutionary advancements in human-computer interaction. The GUI thrust computers forward by creating an entirely new way computers and humans could talk to each other. Spatial data management leveraged the innate and evolved human ability to organize objects physically. And the desktop metaphor provided people with the scaffolding necessary to get started on a computer. – Harbingers (Part 1)
Iterations of the desktop metaphor have defined “real” computers ever since.
What’s a computer?
…to the surprise of many, users very quickly discarded any semblance of indirection [between the icons and the files themselves]. This icon is my file. My file is this icon. One is not a “representation of” or an “interface to” the other. – John Siracusa, About the Finder…
Harbingers began as an exploration of the infamous it’s-not-philosophical-but-it-really-is question posed in a 2018 iPad ad: “what’s a computer?” I became fixated on finding an answer when Dieter Bohn took a swing at it in April of that year, and have toyed with the philosophy of the question ever since.
At risk of spoiling things, this series is called “Harbingers” because I wanted to answer a question of my own: what’s with all the “stage” names in Apple’s marketing lately? Discussed at length in episode 350 of The Talk Show, Gruber elaborated on how inscrutable the names “Center Stage” and “Stage Manager” were, in juxtaposition to one another:
I do think it’s a I think it’s a bad name — not in and of itself — but it is a bad name alongside Center Stage because, in both cases, what the stage is, is entirely different! In Center Stage — the thing with the camera where you can move around in front of your iPad, or your studio display, and the camera pans because it’s ultra wide — the stage is your real world space where your sack of meat with teeth (that is, yourself) is operating. Your kitchen, your office, wherever the hell you are, it is the actual real world where you are in front of a camera. In Stage Manager, the stage is the screen, which could not be more opposite! It is, it is through the looking glass, but they’re calling both of them the stage, and it is very confusing, right?
Now, there’s one more question that’s been floating around as we’ve (perpetually) approached the release of an Apple mixed-reality headset: how the hell will this thing work? There have been no concrete answers to this question, despite a litany of details about forthcoming headset’s hardware – not unusual for an Apple release. Will the headset be gesture-controlled? Will it use a keyboard and mouse? Will it use your iPhone as a controller? What will the interface look like?
In investigating these ideas separately, they happened to collide in this series. When we ask, “how will the headset work?” or “why the mixed use of ‘Stage’ names?”, we’re really asking, “what’s a computer?”
How I learned to love the name
Here’s my bet: the big announcement tomorrow won’t really be the headset. It will be a new paradigm for computing: the Studio metaphor.
When you zoom out and look at Apple’s cutting-edge technologies, the use of stage terminology feels deliberate. Of course, you’ve got Stage Manager and Center Stage. But theater terms show up everywhere in their technologies when you start looking: Focus1. Studio Display. Mac Studio. Spotlight Search. And that’s all before realizing that a “Studio metaphor” solves Gruber’s naming conundrum: if the world is the computer, the “Stage” is both the screen and the sack-of-meat-with-teeth real world space. If Stage Manager and Center Stage are designed for XR, of course the stage is both virtual and reality. It’s so elegant!
The proliferation of the GUI brought new a set of ideas about human-computer interaction when the Mac introduced the mouse. Multitouch broke through when it allowed for a sense of direct manipulation2 of the objects underneath the screen. So it makes sense that physical interaction will bring an entirely new set of design practices to the digital world. Stage Manager, in this view, is the beginning of an answer to these questions. It’s the dawn of a new computing paradigm. It’s a new way to get work done.
In fact, many of the problems Stage Manager is designed to solve are problems that become exacerbated in augmented reality. For example:
- It’s important to remove the feeling of “cleaning up” windows in AR because AR is immersive — it would suck to have both a virtual world and a real world to keep tidy.
- Having visible distractions filling your entire field of view is a great way to get nothing done, so persistent spaces keep you focused on one task at a time. Additionally, single-window mode doesn’t work in MR unless the app itself is a complete immersive experience, so you’ll want options to do windowed work in a way that allows for direct manipulation.
Just look at the set of criteria Craig Federighi gave TechCrunch as important facets of Stage Manager (emphasis mine):
“Building to M1 was critical as well,” says Federighi. “From the start, the iPad has always maintained this extremely high standard for responsiveness and interactivity. That directness of interaction in that every app can respond to every touch instantaneously, as if you are touching the real thing underneath the screen. And I think it’s hard sometimes for people to appreciate the technical constraints involved in achieving that.
“And as you add multiple apps into play, and large amounts of screen real estate, you have to make sure that any one of those apps can respond instantaneously to touch in a way that you don’t have that expectation with a desktop app. Indirect manipulation gives you some slack there, so it’s a different set of constraints.”
Stage Manager takes advantage of the more powerful GPU, faster I/O in virtual memory, faster storage and more RAM that the M1 chips brought to the table.
Notably, Apple backtracked on their policy that Stage Manager would only work on M1 iPads…with the caveat that non-M1 iPads would not support external display connectivity. Craig again:
“We also view Stage Manager as a total experience that involves external display connectivity. And the IO on the M1 supports connectivity that our previous iPads don’t, it can drive 4K, 5K, 6K displays, it can drive them at scaled resolutions. We can’t do that on other iPads.” […] “We really designed Stage Manager to take full advantage of the M1. If you look at the way the apps tilt and shadow and how they animate in and out. To do that at super high frame rates, across very large displays and multiple displays, requires the peak of graphics performance that no one else can deliver.
Wow, lots of talk about large, high-resolution, high-framerate displays. I wonder what that could be useful for?
“When you put all this together, we can’t deliver the full Stage Manager experience on any lesser system,” Federighi says. “I mean, we would love to make it available everywhere we can. But this is what it requires. This is the experience we’re going to carry into the future. We didn’t want to constrain our design to something lesser, we’re setting the benchmark for the future.”
So, in Apple’s eyes, Stage Manager is the window management system “for the future”, and I’d point out that while Federighi’s stated obsession with high resolution, high frame rate, and low latency interaction is important on the iPad, it’s doubly important in a MR environment: you don’t want your interactions with mixed reality to be pixelated, slow, or laggy. These facts only underly our previous convictions about Stage Manager; as far as user experience goes, beyond technical requirements, MR begs for a window management system that affords minimal distraction and minimal clean-up.
On top of all that, a studio metaphor would actually be an extension of the desktop metaphor: desktops can exist in studios! To me, it makes perfect sense that if you’re extending computing to a third dimension, then extending the metaphor to a third dimension makes a lot of sense.
But, hey. It’s 9:46PM on June 4, 2023 as I write this. We’re a mere twelve hours away from the WWDC23 keynote. If a Studio metaphor shows up tomorrow, it would be yet another step in Apple’s history of paradigm-shifting computing metaphors. I think it would be a lot of fun, too. If not, it was, at least, fun pondering while I had the chance. Tomorrow should be fun too.
Thanks for reading, and happy dubdub <3