The B-17 and B-29, Boeing's WWII era airplanes flew at much slower speeds.  The B-17, which had almost no movable surfaces other than the rudder and ailerons, only flew at about 150mph at cruising speed.  The B-29 with its flaps and movable leading edges could get that cruising speed up to about 230.   Assembling those planes on a moving assembly line with the tooling technology of the early 1940s, was not that hard.  But, with the introduction of jet engines on the B-47, the cruising speed jumped to 560mph, with a top speed of well over 600.  At those speeds, even super tiny differences from one side to the other could have a huge impact.  A scuff mark from a shoe in the wrong place on a wing surface could take 10mph off of the speed of the plane.  Less than one degree of difference in the angle of the wings where they joined the body could consume much of the rudder's motion trying to correct it.  Because of these seemingly tiny differences, a plane may look perfectly aligned and still be horribly hard to fly.  Such planes are called bananas, and most large transport aircraft built before the 777 were bananas, to one degree or another.  Because of this, planes have a set of very small control surfaces so the flight crew can adjust its trim.

 

The 777 was a zero trim airplane.  It came out of the factory with enough assembly precision that most of them will fly straight with little to no trim.  The way Boing had achieved this, was over decades of gradually improving what we called monumental tooling.  Massive, two and three story high structures made of steel would hold the major parts of a plane in absolutely the right position while the fasteners were installed.  The foundations for these fixtures, in the Renton plant went over 40 feet deep into the ground.  In Everett where the soil is much more stable, they went down only about 20 feet.  And, we are talking massive concrete foundations with 1" rebar on 12" centers, running in all three dimensions.  Now we were going to convert to doing this assembly on a moving line without any help from such foundations.

 

What made this conversion possible was the combination of laser measurement tools and networked computers controlling the tooling in real time.  Assembling the plane, from a controls point of view, is not all that different from flying it.  You need to have absolute control in all three directions of possible motion.  Except assembly is more difficult than flying by quite a bit.

​

The controls that Wilbur Wright first figured out to individually control the three orientations of a plane (roll, pitch, and yaw) are for managing the orientation of a single object with respect to the flow of the air over its surfaces and its forward movement through the air.  A plane in flight is a single object needing control.  When assembling a plane, multiple parts have to be controlled with respect to each other.  With computers and lasers we knew we could do this without those massive monuments.  But, the tooling control system and its networking is critical.  In this respect, a tooling network is very much like the flight control network on the plane.  It's worth noting that in the factory we talk about "flying" the major airplane sections together, and the major crane moves which happen on third shift make parts of this process actually look a bit that way.

​

The 777 program not only wanted to build this sort of network control system to enable them to fly the plane together just as accurately with the moving line tooling as they had with the monuments, they wanted to lash all of the controls for this bit of acrobatics together over the general purpose network service provided by IT.  If that wasn't a tall enough order, they also wanted many of the key links to be wireless.  How to do this safely was the essence of the specification that Mike had asked me to write.

​

To get a visceral feel for what Mike wanted, ask yourself this:  Would you fly in an airplane whose flight controls were exposed to the public internet?

​

It wasn't that hard, and it only took me about three screens worth of an email message to crank it out.  The big thing was that the control system needed to have a lot of internal security and communications accuracy checking not normally found in personal computers.   The components or nodes in the control system had to treat the Boeing internal network as being every bit as potentially hostile as the outside internet.  Assuming that something provided by the IT organization was hostile was right up my alley.  That was phone call number one.  As I recall, it came in sometime around January of 2004.  It was one of those routine things and I could be off by a month or more, maybe even a year.  This is a great example of the kind of routine consulting work I was doing after moving from AR&T to BCA.

 

Phone call #2 from Mike came in on the 16th of December 2004.  It was a Thursday afternoon, late in the day.  It was only two days from when the traditional Boeing holiday break would occur.  The day was overcast, and moderately cool.  It was one of those days when something happens that you just never forget.  I know exactly where I was sitting (I had a couple of dozen office locations in my years at Boeing), and what I was thinking about that day.  I was mostly mulling over the kid's Christmas presents that I still needed to get. 

​

Mike told me that the contractor hired to build the larger equipment for the 777 moving line had ignored my specification, and built the control system like it was a regular collection of personal computers.  The first pieces of equipment were going to arrive in the factory during the break, and there was too much at stake not to use them.  How were we going to make this work?  There is only one way to describe a moment like this.  Oh shit!

​

The Boeing 777 was the most profitable product we had, and the biggest source of revenue in the entire corporation.  Tens of thousands of jobs were directly dependent on it.  It was a critical part of the global transportation system.  Now the program was in jeopardy in a way that the top leadership of the company would never understand nor even know about.  And, the problem was on my desk.

​

There were two obvious answers, and one terrible one that IT would suggest if asked.  One was to give up on the idea of using the BEN, and quickly build a dedicated factory network that was air gapped from it.  That was doable, but it was expensive.  I would have a hard time selling the need for it to managers in IT who knew nothing about the technology, but I could roll over them by getting BCA management to dictate it.  But the bigger problem was that such an approach was super risky. 

 

Keeping two large industrial networks air gapped is an almost impossible task.  The temptation to install a potential router, which can be as simple as a laptop computer with two network interfaces, ... well, that temptation to join any two networks is huge.  The odds were extreme that someone would take it upon themselves to do just that in order to solve some problem of moving data around the company.   And even if the two networks were not permanently connected by someone, just moving a laptop back and forth between them to become a conduit for all sorts of malware.  At Boeing, the technical community is (or was) filled with action oriented technical people.  If a problem needed solving, it would just get done.  T. Wilson had once jokingly referred to the Boeing technical workforce as the squirrels.   Problems were like acorns.  Throw them to the squirrels and let them go to work.  The squirrels will just make the problems disappear.  A separate network, however appealing on the surface, was a bad idea.  That said, the alternative was hugely difficult.

​

The bad idea that the IT networking folks would jump to involved manual configuration management of a solution operated by the IT networking folks.  This approach would require lengthy service requests every time a tooling engineer needed to make a configuration change on the line, which might be as simple as swapping out a failed piece of equipment with a spare part that had a different serialized network identifier.  [Except in cellular telphony, but sometimes even then, these serialized identifiers are typically called a Media Access Control ID or MAC, which is an identifier, not an address; no matter how many text books and network engineers say otherwise.  An address is a location in a hierarchical system.  That is not what a MAC is.] 

​

This other option was another one of those big hairy audacious goal things: fix one of the longstanding flaws in TCP/IP networking.  While this approach was audacious, Boeing had a solid history of tackling core internet architectural challenges (see "Boeing and the Internet").  But was there time?  I needed help.

​

Steve Venema is hands down the smartest and most capable network technology researcher and engineer I have ever met.  Steve had a lab in AR&T where he worked on some really interesting projects.  I called Steve and he was eager to help.  He had already done several projects for Mike Callaghan in the Everett factory.  I got back to Mike Kozak and he setup a meeting for all of the key technical people to get together in an office building we were leasing in Lynnwood in early January.  These were the same offices that had hosted the 747 FAIT program.

​

It turned out the the problem of factory networking I was sitting on was very similar to networking on the battlefield.  Steve had gotten a contract from the Marine Corps a couple of years earlier to come up with a way to enable the Marines to quickly setup a battlefield network that was safe from hacking by an adversary.  He had been wanting to do some more work on this project and take it to the next level.  The 777 moving line networking challenge was a perfect fit for what he already wanted to do.

​

Before continuing, let's talk a bit about technology and ethics - you know, that stuff that separates right from wrong - the stuff that seems totally lost on most of the tech industry.  We're talking about very basic stuff like we try to teach our children in kindergarten.  This applies to everyone, everywhere.

​

Here is a modest proposal. 

  1.  When someone owns a computer and connects it to a network they should be able to easily look and see what the source is of any packet, however small and trivial, that arrives at their computer.  Think of the computer as an extension of your body.  As my daughter put it so succinctly when she was about three: "I'm the boss of me."  It is simply wrong to go messing around on someone else's computer without their explicit and informed knowledge and consent.  If they don't understand what you are doing, then getting their consent is meaningless.  Proceeding anyway is no different than assault or trespassing, and yet doing so has become a standard part of the basic business models of most tech companies.  It's disgustingly awful, and should be illegal.  The reason that it isn't illegal is because our elected officials are fair game for K Street, and even the best of them have been routinely conned.

​

  2.  If someone doesn't want anyone except people, or companies, or other computers that they specify to be able to send a packet to their machine, that should be easy to do.  Again, "I'm the boss of me."

​

  3.  If they don't even want their network service provider to know about what controls they have setup, that also should be easy to do.  Show me a carrier that understands and respects their customers' privacy - go ahead - find just one such carrier.  I dare you.  Repeat: "I'm the boss of me."

​

That's pretty simple.  It's really the basics of everyday etiquette expressed in networking systems.  But, as things stand right now in 2022, the internet is inherently invasive.  We are blind to most (not just some, but most!!!) of the traffic in and out of our machines.  It's just awful.  It's pretty close to just plain evil.  But, the utility of networking is so great, and we feel so small and insignificant in the face of how big it is, that we tend to just accept it.  It's as though we are a bunch of early medieval peasants trying to avoid the inquisition.  I personally think this has to change and a few tech CEOs need to get some very long prison sentences, but set that aside for a moment. 

 

When it comes to control systems which are critical to the safe operation of everything that matters in our world, this kind of lack of security is a complete non-starter.  Control systems are everywhere from the flight controls of airplanes, to the operating systems of nuclear power plants (remember Three Mile Island - OK, I'm a bit older), to the traffic lights on our streets and highways.  It's nice for the people responsible for the operation of these things to have networked remoted controls and monitors.  It cuts way down on operational costs and makes their work dramatically easier.  But, anyone not explicitly authorized should not be able either detect their existence or send a packet to identify them via the internet, even if they are equipped with the most sophisticated network sniffing equipment available.  As the saying goes in network security circles: "if you can't see it, you can't hack it."

​

The rampant lack of ethics associated with the business models of the entire technology and venture capitalist parts of the economy are simply intolerable when life is at stake.  Steve had figured out a fix to these problems.

​

Steve's initial solution involved using an invention by Bob Moscowitz from Chrysler.  It is called the Host Identity Protocol, or HIP.  Then along the way of building the 777 solution, an interesting accident occurred that made Steve's solution many times better than his team's implementation of HIP alone.

​

One day my friend Pete Fayette called me.  Infoblox, the company that provided some of our network management equipment and software, wanted to come to Seattle and give Boeing an award as being one of their key customers.  Pete wanted to know if I could give them a tour of the Everett factory.  The folks from Infoblox were particularly interested in any networking challenges we might have.  Manna from heaven!  I reserved a conference room for a before and after tour meeting.  There, Stu Bailey and his team heard about our 777 moving line challenges, and I walked him and his team through about half of the factory, showing them a pretty good variety of the tooling and facilities control systems.  Afterward, Stu told us about a project he was leading at the Trusted Computing Group.  We upped Boeing's level of membership in TCG and Steve's team became participants in the continuing evolution of TCG's IF-MAP protocols, later renamed TNC-Grid at the insistence of Cisco.  This was so their advertising people could confuse things with another product they were selling (see my eyes rolling about tech ethics again).  When the IF-MAP stuff was combined with HIP, the whole 777 problem became dramatically easier for everyone involved - except IT of course.

​

Customer enablement and self-facilitation are often anathema to IT managers more concerned about the boundaries of their empires.  IT priorities always trumped the costs and efficiencies of the supported organizations, since none of that stuff showed up in IT's performance metrics.  Boeing's IT leadership, true to form, failed to understand why this stuff was so important, so it eventually walked out the door.  Dave Mattis who worked with Steve, left Boeing and used our 777 moving line networking solution as the starting point for what is now the company Tempered Networks.  Orly Brewer, another key member of Steve's team left to join Dave as well.  Later, Steve would get fed up and leave Boeing as well.  I'll say a little bit more about this technology in the section "Boeing and the Internet."

​

Another networking project that I worked on a little bit in the factory deserves a brief mention here.  That was how to manage the systems that store the software that gets loaded onto the planes during production.  Loading that software safely, required me to learn a whole lot about the internal network architecture of Boeing airplanes.  That included both cabin systems, maintenance controls, and flight controls.   

​

While flight controls are well outside my areas of expertise, I am very familiar with all of the principles and most of the components involved.  I grew up around flying.  My dad had been a Navy Commander, and pilot in maritime patrol.  That was in the days of the PB4Y2 Privateer and the P-2 Neptune, predecessors to the P-3 I had worked on, and the Boeing P-8 Poseidon where I consulted on the IT support systems for a few months.  Later, dad became a pilot for a company that is now buried deep inside United Technologies, Corp.  My introduction to planes and flying came very early.  The first time I had partial control of one in the right seat I was only ten and still couldn't reach the rudder pedals.  But dad flipped over the wheel and there I was in control of two of the three basic movable surface systems of a Beech Bonanza.  I never did go through the process of getting my pilot's license, but I have some confidence that I could fly and even do a decent landing in clam air.  So this stuff was in my blood as much as accounting, computer networks, history, or woodworking.

​

What you can see from all of this is how I ended up learning a huge amount about how we build airplanes, and how the control systems associated with both the factory and the planes are designed and assembled.