PC Pro

Taking computers to the NeXT level

Researcher­s believe AI-powered language tools could be turned from human to chat to decipherin­g dog barks. Animal lover Nicole Kobie reveals all

When Steve Jobs left Apple he hoped to make a big impact with computers for education. But NeXT ended up teaching the computer industry a thing or two instead, as David Crookes explains

In summer 1985, Apple co-founder Steve Jobs rebuffed three offers to become a professor. “I told all of the universiti­es that I thought I would be an awful professor,” he later revealed in an interview with Newsweek. Yet he still wanted to make a big impact in the education sector.

Jobs was 30 years old and ready for a new challenge. He’d resigned from Apple following a reorganisa­tion, but not before telling the board what he wanted to do next. Having previously visited Brown University, he’d been told academics sorely wanted a powerful, personal workstatio­n capable of helping them with their research. His aim, he told the Apple board, was to create a computer for the higher education market that best suited them.

When he made his announceme­nt, jaws dropped, primarily because he said he was going to take five Apple employees with him. “These are very low-level people that you won’t miss, and they will be leaving anyway,” Jobs explained. But he wasn’t entirely telling the truth.

Jobs had little love for those running Apple at the time. “I think John [Scully, Apple’s then-CEO] felt that after the reorganisa­tion, it was important for me to not be at Apple for him to accomplish what he wanted to accomplish,” Jobs told Newsweek. “He issued a public statement that there was no role for me there then or in the future, or in the foreseeabl­e future. And that was about as black and white as you need to make things.”

To that end, he didn’t care that the five people he chose for his new venture were hugely important to Apple. Among them was Rich Page, one of the first four Apple fellows who had prototyped the company’s first portable, colour and 68020-based Macs. At the time, Page had been working on a machine referred to as the Big Mac, a powerful workstatio­n for use in a university lab. It was a dual-boot machine allowing for the UNIX operating system as well as Mac OS but ended up being cancelled. The five also included Bud Tribble, who headed Apple’s software developmen­t team and helped to design the classic Mac OS and its user interface; engineer George Crow, who designed the Macintosh 128K’s analogue board containing the power supply and video circuitry; and Susan Barnes, controller of the Macintosh division. Then there was Dan’l Lewin, a sales executive in the Mac team who created the Apple University Consortium and bulk sold Macs to two dozen institutio­ns, including all Ivy League universiti­es. All were disillusio­ned with life at Apple.

In that respect, Jobs was being honest. His chosen few likely would have left regardless of his intention to create afresh. Jobs didn’t waste much time, either. “Steve called me and I went for a long walk with him,” Lewin told PC Pro. “He told me he was going to start a company and it was going to focus on the things that I cared about – the things I’d pioneered at Apple from scratch.

“My thought process was that if I didn’t go and the company was successful, I’d kick myself, and if I didn’t go and it failed, I’d be curious as to whether or not my participat­ion would have made a difference. At that time, I was running the entire education side of Apple, which was two-thirds of the company’s revenue – I’d built both a distributi­on strategy and a market framework for the higher education and research community. But I was a

“Jobs didn’t care that the five people he chose for his new venture were hugely important to Apple”

little frustrated by what was going on inside of Apple.”

Lewin was upset that Apple’s direct sales to universiti­es had been affected by a reorganisa­tion and the fact that product groups had stopped sharing informatio­n, which made his work more difficult. “One of the products which mattered to me was Big Mac,” he added. “I didn’t feel like I could trust getting the things done that needed to be done at Apple.”

Jobs’ new company offered a way forward and Lewin climbed aboard.

The next step

Apple was furious. Bill Campbell, then VP of marketing, was incensed that Jobs had hired Lewin because of the strong relationsh­ips he’d built with universiti­es. Apple threatened to sue Jobs, who told the press he found the situation strange. “It’s hard to think that a $2 billion company with 4,300 employees couldn’t compete with six people in blue jeans,” he said.

Yet with Lewin and the others on board, it didn’t really matter that the new company – set to be called NeXT – was short on numbers. The team’s expertise would be enough to make an impact. “We have an immense amount of confidence in each others’ abilities and genuinely like each other,” Jobs told Newsweek. “And all have a desire to have a small company where we can influence its destiny and have a really fun place to work.”

It helped that the groundwork was already laid. Lewin’s network ran deep and he had a great understand­ing of the computing needs of higher education. “It had a reputation of being a niche market,” he explained, “but I turned that inside out and said, actually, it’s a superset of the real world.

“You have consumers who are replenishe­d annually. You have researcher­s doing the most far out, extensive research on the planet. And you have these incredibly bright grad students who are pushing the edge of the envelope – they were doing fascinatin­g, fantastic things with computers and foundation resources were funding their work. Major research institutio­ns in the US are billion-dollar operations complete with this unique set of characteri­stics.”

As such, NeXT was confident that it could corner the market simply because the team knew what it would take. “We’d looked at what was going on in higher education and in primary research institutio­ns and noted that they were using VAX machines, occasional­ly an Apollo workstatio­n and, more and more over time, Sun Microsyste­ms’ workstatio­n.

“But any researcher worth their salt had a budget of a quarter to half-a-million dollars to gain access to a VAX, so we were looking at what they were doing with those machines. It led us to thinking about the architectu­re of the computer we wanted to build.”

Seeking perfection

It took a long time for the NeXT computer to be developed. Even in late 1986, as noted in Walter

Isaacson’s biography of Steve Jobs, the company only really had a $100,000 logo created by the American artist and graphic designer Paul Rand and some snazzy offices to show for their endeavours. “It had no revenue or products,” Isaacson wrote. But that’s not to say it didn’t have anything of note. NeXT had secured a deal with Oxford University Press to bundle a digital edition of Shakespear­e’s works with the machine, alongside the Oxford Dictionary of Quotations, a dictionary and a thesaurus – thereby inventing the idea of searchable ebooks. NeXT had also nailed a deal with Lotus to develop a spreadshee­t app for the NeXT’s operating system and enlisted other PC software companies to help.

NeXT had also hired Hartmut Esslinger. He was a master industrial designer who was being paid $2 million a year to create a design strategy for Apple. Jobs persuaded Esslinger to wind down his contract with Apple and set him to work on the creation of a cube case with perfect 90-degree angles and foot-long sides, even though it caused production problems later down the line.

Again, as Isaacson wrote, it’s difficult to get precise cubes out of tight moulds. The solution was moulds with separately created sides, but it added $650,000 to the manufactur­ing costs. Jobs’ penchant for perfection also prompted the purchase of a new sanding machine. The screws inside the NeXT were plated as well, further inflating the costs. But at least the NeXT was moving forward.

“One of the design points was putting a digital signal processor in every machine, which no-one had ever done,” said Lewin, discussing the more important matter of what was going to go inside that cube. He was talking about the Motorola 56001 DSP, which allowed for fast processing of large matrix calculatio­ns, enabling the generation of CD-quality sound, speech, tone detection and music.

The NeXT computer would also include a Motorola 68030 processor running at 25MHz, built-in Ethernet, a high-resolution display and, crucially, a 256MB magneto-optical storage medium. “That was a lot of storage at the time,” Lewin told us.

“It was also erasable, removable and non-magnetic.”

Powering up

One thing was clear: the team wanted to approach the higher education market in a different way. “We wanted to look at the personal computer sales and distributi­on model and the workstatio­n applicatio­n market opportunit­ies,” Lewin said. “At the time, when you bought a PC the operating system came with it and you’d only pay for upgrades. You didn’t pay any monthly fees.

“But in the workstatio­n business, Sun Microsyste­ms being the dominant player, you’d pay a monthly fee to keep the operating system alive. So we wanted the power of a workstatio­n machine with a PC business model, bundling Unix at the core. We wanted to dismantle the distributi­on side of the workstatio­n market.”

Lewin knew that universiti­es didn’t relish having to build power plants to run workstatio­ns. “That

“It’s hard to think that a $2 billion company with 4,300 employees couldn’t compete with six people in blue jeans”

happened at Berkeley when they got 1,000 Sun 1 workstatio­ns,” he said. “Universiti­es also had to pay system engineers to move computers down the hall because if they physically moved them, they’d violate their terms of agreement.”

The NeXT Computer would need 300W of power. “We could help create large labs,” Lewin said. On paper, it seemed like a perfect machine which, aligned with NeXTSTEP, should have been a winner.

Groundbrea­king OS

NeXTSTEP was an object-oriented, multitaski­ng Unix operating system based on the Mach kernel. “The distinguis­hing characteri­stic, at least in my mind, was the choice of Objective C and dynamic runtime binding,” Lewin said. “You could imagine professors building software and wanting to introduce an object into a simulation of some sort, looking at the reaction and checking what changed, and that’s what happened.”

But the graphical mouse-based operating system was hugely innovative in other ways. It introduced the idea of the Dock seen in modern Macs. It had a 3D-style interface, high-resolution icons, real-time scrolling and window dragging, fluid graphics rendering and built-in networking support. “Networking is pretty fundamenta­l to Unix and also where the world was heading,” Lewin explained.

A few years later, NeXTSTEP also ended up giving the world what could be described as the first digital App Store courtesy of the Electronic App Wrapper, a commercial electronic software distributi­on catalogue that allowed users to purchase, decrypt and install apps automatica­lly. It’s just a shame it took so long for it to see the light of day.

Jobs had wanted the NeXT Computer to be released with the operating system within 18 months. That was never going to happen, but you couldn’t fault the ambition and attempt to push developmen­t along. While work continued, there was investment – notably from Ross Perot, the founder and CEO of Electronic Data Systems and Perot Systems.

Yet there were also pushbacks, primarily from Microsoft founder

Bill Gates, who went as far as to call the machine “crap”, stating “the optical disc has too low latency”.

But there was beef between Jobs and Gates. Jobs wanted Microsoft to create software for the NeXT Computer but Gates didn’t want to go in that direction. There was an attempt to license NeXTSTEP to IBM instead and create a de facto rival to Windows, but that potential deal collapsed.

In the end, Jobs unveiled the machine before 3,000 people in San Francisco’s Davies Symphony Hall on 12 October 1988, some three years after leaving Apple. It came with the NeXTSTEP operating system as a machine exclusive, and Jobs, who had invested $12 million in the company, was in a confident mood as he showed off the machine’s ability.

It could quickly retrieve a line from Shakespear­e, he demonstrat­ed, and capably play a duet with a real violinist. Newsweek said the black cube “may be the most exciting computer in years”, and the presentati­on earned a standing ovation. Yet the computer was a pricey little thing. Jobs wanted the computer to cost no more than $3,000 when it was under developmen­t. It ended up selling for $6,500, and that was with a university discount.

If buyers wanted a printer, that would cost an extra $2,000. They’d also need to wait for version 1.0 of NeXTSTEP, which wasn’t released until 1989. But the fact that the company didn’t give up was admirable, and Jobs had a response to those who reckoned the computer was late. He told them it was actually five years ahead of its time.

The legacy

Even so, Lewin soon became concerned. Although the NeXTcube and NeXTstatio­n were released in 1990, the company wasn’t in good financial health. “We had $120 million in the bank and I could see that being burned within 12 months, and that’s exactly what happened,” he said, resigning in 1991. “But in the end the asset was always software.” Other variations followed, until NeXT decided to concentrat­e on NeXTSTEP in 1993.

The company would go on to leave a strong legacy. British computer scientist Tim Berners

Lee created the world’s first web server and web browser on a

NeXT computer at CERN, hosting the first web page in December 1990. The video games Doom and Quake were also created on the platform – developer John Carmack said id Software bought its first NeXT out of personal interest and ended up spending $100,000 on the machines.

“It is funny to look back; I can remember honestly wondering what the advantages of a real multi-process developmen­t environmen­t would be over the DOS and older Apple environmen­ts that we were using,” Carmack wrote on Quora in 2016. “Actually, using the NeXT was an eye-opener, and it was quickly clear to me that it had a lot of tangible advantages for us, so we moved everything but pixel art (which was still done in Deluxe Paint on DOS) over.”

NeXT’s biggest legacy, however, involved Apple. In December 1996, Apple acquired NeXT for $427 million in cash, shares, stock options and debt. Apple wanted NeXTSTEP to become the foundation for a new Mac operating system to replace classic Mac OS. The move also heralded

Jobs’ return to the company.

It meant Jobs went full circle, proving he was still able to influence Apple’s direction even when not working at the company. “If you look at the Mac today, you can trace the operating system right back to NeXTSTEP,” Lewin said. There’s the Dock, many base APIs, the Objective C runtime, the manner in which thirdparty apps work, and so much more. So while NeXT didn’t prove to be hugely successful as an entity in and of itself, it had an immense impact on computing as a whole. And far from having an adverse impact on Apple, it played a large role in being its saviour. In that sense it was a successful failure, one that ultimately ended up making Jobs a household name. “It was,” Lewin said, with a level of understate­ment, “an interestin­g moment in time.”

“Jobs had a response to those who reckoned the computer was late. He told them it was actually five years ahead of its time”

Woof, woof. You stare at your dog’s fluffy face and wonder two things: what does she want, and how do I make her stop? AI researcher­s are working on a way to answer that first question so you can then figure out the second, following a grand tradition of attempting to unpick what our pets are trying to say to us.

In 2002, Japanese company Takeda launched the BowLingual dog translator ( tinyurl.com/361bowling­ual), followed by the Meowlingua­l version for feline friends. By listening to the sounds your pets were making, this $75 device would supposedly reveal its mood via an onscreen display and a short phrase (think “I’m hungry”). It was awarded that year’s satirical Ig Nobel prize, but animal translatio­n using AI has attracted real academic attention.

A project called Zoolingua is developing an app to translate dog body language as well as building a wider data set. The Georgia Institute of Technology has been tracking chicken sounds in different situations – such as hot or cold – to understand which conditions the birds prefer. Largely successful­ly, it

would seem. The Project Cetacean Translatio­n Initiative is using AI to unpick what sperm whales are saying, while the charity Wild Dolphin Project has used machine learning to try to understand dolphin speech – both of which require us to be able to think like a dolphin or sperm whale, as the New Yorker noted ( tinyurl.com/361dolphin).

So people are taking this seriously – but others aren’t. A Berlin-based PR agency unveiled a new product last year called, of course, BarkGPT: “a never-before-seen web and app-based tool that uses natural language processing and machine learning to translate recordings of a dog’s barks into human language.” While that marketing copy sounds exactly like the claims made by a good chunk of pet translatio­n apps available on the mobile marketplac­es, this particular example was an April Fool’s prank.

It may not have seemed funny to researcher­s at the University of Michigan and Mexico’s National Institute of Astrophysi­cs, Optics and Electronic­s (INAOE) Institute, who applied models trained on human speech to animal communicat­ion – and they think they’ve made progress.

Decoding dogs

One of the project researcher­s, Artem Abzaliev, told PC Pro the project came about because he and his advisor wanted to apply a known type of language system to an unknown one, just to see if it would work. “My background is in visual language processing, or I guess people call it AI nowadays, but basically text understand­ing,” he said, such as reading a film review and understand­ing that the writer enjoyed the film.

Why dogs? We don’t know what they’re saying, so the idea was to take a system that we know works on humans and see if it could be applied to something else that has speech but isn’t yet deciphered. Plus, Abzaliev likes dogs.

Beyond that, this work also addresses a core challenge with applying AI to pet communicat­ion: a lack of training data.

For wild animals, such sounds need to be collected in the wild. Pets are easier to find, but permission is needed from owners and it’s time consuming.

Because of this lack of training material, analysing dog vocalisati­ons using AI has long proved difficult, despite the many apps and efforts listed above. The researcher­s wanted to see if existing human language models could be applied to dog barks, after being tweaked to work with animals, of course. After all, plenty of work has been put in to understand all aspects of our speech, as that’s what allows voice assistants to work.

The work required a fresh dataset of dog vocalisati­ons, which a team at INAOE collected from 74 different dogs across a range of ages – there were puppies as young as five months – and a selection of breeds popular in Mexico, where the recordings were made. That included 42 Chihuahuas, 21 French poodles and 11 Schnauzers. The dogs were filmed reacting to various stimuli, such as their owner returning home, playing and being introduced to a stranger, resulting in the developmen­t of 14 different categories of vocalisati­on, including a “positive squeal” and a “negative grunt”.

That data was then used to tweak a speech representa­tion model known as Wav2Vec2, which was originally trained on 960 hours of human speech, looking at tone, pitch and accent in particular.

Did it work?

Modifying that existing model seems to have worked to an extent, with accuracy up to 70% on one of four classifica­tion tasks. To be clear, the aim wasn’t to determine what the dog was trying to “say”. Instead, researcher­s were trying to identify specific dogs, their breed and gender, whilst exploring the impact of context (a concept called grounding).

For dog recognitio­n – trying to pick out which individual dog was “talking” – the Wav2Vec2 system scored 24% when used without further training, and 50% after pretrainin­g with the dog data. Picking out breeds was trickier: the pre-trained system had a 75% accuracy rate with Chihuahuas, but that fell to 36% for poodles and 15% for schnauzers, for an overall rate of 62%. “That’s not too bad,” Abzaliev said.

But for gender, the system didn’t work. “We basically do as good as random,” explained Abzaliev.

“Gender prediction turned out to be very hard. It’s hard to do even in humans, but for dogs, even harder.”

The last task, grounding, looks at the connection between a dog bark and its surroundin­gs, perhaps the closest classifica­tion task to understand­ing what a pet is trying to communicat­e. Here, the Wav2Vec2 system pretrained on dog sounds scored 62% accuracy.

The work raises a question: how do you even know if a dog bark translator is accurate? It’s not as though the dog can correct the system the way we can with human speech models. “We don’t,” admitted Abzaliev. “And it’s not exactly accurate.”

Why it works – when it does

The results may not sound too impressive, but it’s a leap above other models trained only on dog barks, suggesting the idea of applying existing language models to animals might be sound.

“This is the first time that techniques optimised for human speech have been built upon to help with the decoding of animal communicat­ion,” said Rada Mihalcea, the Janice M Jenkins collegiate professor of computer science and engineerin­g, and director of UM’s AI Laboratory, in a statement.

“The idea was to take a system that we know works on humans and see if it could be applied to something else”

“Our results show that the sounds and patterns derived from human speech can serve as a foundation for analysing and understand­ing the acoustic patterns of other sounds, such as animal vocalisati­ons.”

It may seem surprising that dog vocalisati­ons are similar enough to human speech for this to work – and Abzaliev admits the researcher­s weren’t themselves convinced that a language model developed for understand­ing human speech would work on animals. “Our expectatio­n was that it shouldn’t work,” he said. “Dog barks are very dissimilar to human language. It shouldn’t have worked... It didn’t help completely, but it’s interestin­g that it helped.”

Indeed, the team isn’t even sure why applying a human speech model to dog barks helped at all, if even in a limited way. “We’ve had a lot of hypotheses trying to understand,” Abzaliev said, adding that the improvemen­t in accuracy in their research over previous efforts that didn’t use a pretrained model could be down to the fact that their own system learns how to hear or listen better.

He adds that it’s not unheard of in other areas of AI. “Similar things happened in computer vision as well,” Abzaliev said. “For instance, pretrainin­g on... one completely or relatively unrelated domain helps.”

Knowing something is better than nothing, it would seem. Even when it comes to artificial intelligen­ce.

How accurate?

While the results were successful in proving that repurposin­g existing language models has merit, how could the accuracy be further improved? More training data. This is AI after all, and AI always wants more data. “This is the biggest problem in animal communicat­ions,” Abzaliev told us. “Generally, there’s very little data.”

Large language models such as OpenAI’s GPT have plenty of written text to examine – the entire internet and more – and they still need more data to improve accuracy. But we don’t have a large dataset of dog speech; this project had samples from a mere 74 dogs. “It will need to be manually collected and manually annotated,” said Abzaliev. “It’s a laborious and time-consuming process. Data is a big, big deal.”

How can dog owners help? If you upload dog videos or sounds online, write a good caption so researcher­s can make use of the data. “My dog is barking because it’s happy, my cat is meowing because it’s upset,” is one example suggested by Abzaliev. “If you truly know that’s what’s happening, this would really help everyone.”

What’s next?

There is, of course, more work to be done – this research simply showed that one language model could be applied to a different topic. The paper notes that other neural network architectu­res beyond Wav2Vec2 might be worth trialling, too.

According to the paper, further work is being considered around marine mammals and birds – which both have larger data sets available – and Abzaliev says the project is potentiall­y working with capuchin monkeys. “They have a larger vocabulary size than dogs,” he explained, adding that issues around working with animals remain: “But there is also a relatively small data set.”

Abzaliev is also working on ways to convert the audio files themselves, so specific sounds can be more easily recognised in word-like ways. “Let’s take ‘I love dogs’ – there are three different words,” he said. “But we don’t even know where the words are in the bark or vocalisati­on.”

He added: “So I’m currently exploring models that do tokenisati­on for human language from audio for animal language. We’ll see if it works.”

What’s the point?

Why do we need to know what a dog means when it barks or whines? Beyond better understand­ing the nature of AI language models and how they apply across languages, the UM researcher­s’ aim was improving animal welfare: if we can read more nuance into a dog’s bark, perhaps we can better understand its needs. This could perhaps prevent negative behaviour or health issues.

“There is so much we don’t yet know about the animals that share this world with us. Advances in AI can be used to revolution­ise our understand­ing of animal communicat­ion, and our findings suggest that we may not have to start from scratch,” said Mihalcea.

Abzaliev believes this work could help dogs themselves live happier lives. “I hope it can understand when your dog has anxiety,” he said. “I believe that not everyone understand­s their dogs very well. They have problems understand­ing ‘why my dog is barking’ or ‘why my dog is crying’ or ‘why my dog behaves in a specific way’. So hopefully this can help – we are pretty far away from this, but I hope that it has at least potential.”

To be clear, none of this means you’ll be able to talk to your dog via an app any time soon, if ever. That would require a way to decode human speech into barks, and that’s much more challengin­g than translatin­g French to German. Instead, this early research is more about exploring how AI models can be applied to new tasks – and it seems that existing human language models can be applied to animal communicat­ion, though additional training using data specific to the animal in question improves accuracy. And that understand­ing should help those hoping to better understand chickens, dolphins and sperm whales.

So existing AI models can help speed along animal research, but a dog-to-person translator remains elusive. In the meantime, you’ll need to put the time in to figure your dog out yourself – give your dog a treat, a scratch behind the ears and take it for a walk. We don’t need AI to reveal that will likely make both owner and pooch happy.

“Existing AI models can help speed along animal research, but a dog-toperson translator remains elusive”