The Abnormal Distribution

Martyn Thomas chaired a committee convened by the UK Royal Academy of Engineering on infrastructure vulnerabilities to GPS disturbances. The committee reported in March 2011 and Martyn was briefly on the front page of UK news media on March 10, 2011 until the Tohoku event happened the day after.

What Martyn’s committee found was astonishing. For example, critical infrastructure functions whose builders and operators were convinced had no connection with any GPS functionality – and which stopped working when a GPS jammer was activated. The Committee’s report is well worth reading all the way through. Its remit includes all SatNav systems, not just GPS.

Martyn gave a Keynote talk at the 20th Safety-Critical Systems Symposium in Bristol a couple of weeks ago. A Google preview of Martyn’s paper is available, as well as an IET.tv film of his talk. (The Institution of Engineering and Technology, IET, filmed many of the presentations. You can check out my Keynote on the Fukushima Daiichi accident as well if you like )

It is amazing to me that anyone wouldn’t take Martyn’s observations very seriously indeed.

However, we do appear to have a few journalists that poo-poo it, for example Lewis Page again recently in The Register after his commentary a year ago upon the report’s release, just as we had an astonishing number of journalists who made public their opinion that Y2K was never a big deal. A very silly point of view. As Martyn points out in his talk, the reason Y2K was not a big deal is that people such as himself worked very hard to eliminate as many as possible of the Y2K vulnerabilities discovered in our critical infrastructure, and were obviously quite successful. He knows what they were, since he was the senior technical advisor for some of that work (for example, UK air traffic services provision), and knows what would have happened had they not been taken care of.

The main social point here is, I think, people who worry versus people who don’t. If we didn’t have people who worried, then we wouldn’t be able to operate because things would be continually going wrong, such as possibly UK air traffic services at the turn of the millenium had NATS not worked very hard to eliminate those vulnerabilities. And on the back of such successful effort there are journalists who say “everything’s OK, isn’t it? Why worry?”. Yes, things are OK. Why worry? Because if some of us didn’t, they wouldn’t be.

Here is an example of a daily vulnerability that bit. It’s also old hat. But it happened to me two days ago, and most of those involved are a professional computer scientists with a PhD (or about to obtain one) and decades of experience of such matters.

I have used my e-mail system as a memo system very effectively for the last few decades. I am based on IMAP, so it’s what people now call “in the cloud” but used to be called “stored on a server“. Over the years, when a subject or task occurs to me, I have got pretty good at remembering the context in which it occurred and indexing into e-mail (I send quite a few messages just to myself). It works for me very well. For decades.

Until Tuesday. I was writing an email, and the longish memo I was writing started losing characters backwards from where I had been typing, at the similar repetitive rate to that deriving from, say, a stuck delete key. It took a few seconds to realise what was happening. Then I went into the menu-strip at the top of the screen (I use the Apple OS+environment) and tried to quit my mail client (Thunderbird – Apple Mail apparently does not work well with IMAP. I lost all my mail for about a year at one point a few years ago and it took a couple of days to generate a solution from backup. The second time it happened, I switched to Thunderbird). The menu would come down, but disappeared again as I moved the mouse onto it. This happened repeatedly. I tried the same on the Apple main menu (so I could “Force Quit” the mail client) but the same happened there. I tried a hardware shutdown – the OS refused because Thunderbird would not quit and it advised me to quit Thunderbird and then try again. I have never actually tried to log in as root and am not sure I remember the root password, so trying that, and if successful getting the process number and performing “kill -9” didn’t seem like a good option given the urgency.

So, hardware kill: press the “off” switch and hold until the machine powers down. Good news for me: this worked.

When it came back up and I fired up the mail client, it showed me that all the messages from Wednesday 15 February at 16:35 (15:35 UTC) until that Tuesday morning, 21 February, were no longer there. There are a bunch of important interventions that had disappeared.

So I asked the faculty computer services to restore the mails from backup. One of the two officers is Jan Sanders, with whom I have worked closely for over a decade; he also works with Causalis (people from SSS2012 may remember him from the booth) and will shortly finish his Ph.D. with me. And he installed and maintains this blogging system. These two people, along with 50-75% more help from assistants, manage the Technology Faculty’s (TechFak) computer systems, which account for over half the data volume per day of the entire university. A couple of years ago, we purchased backup hardware for some €30,000 because the university computer center proved to be unable to provide backup services as needed by some high-data-volume colleagues. The university is trying to centralise as many “routine” computing services as possible, and this situation was and is a major negotiating point over the future organisation of research computing services in the university.

Well, our backup HW+SW didn’t work. Jan + colleagues were unable to extract my e-mail Inbox directory alone. They ended up rebuilding the entire TechFak mail-server IMAP file system on a restore disk, some seven hundred gigabytes or so to be restored from main+incremental backup tapes. Estimate on Tuesday lunchtime was Wednesday morning. But on Wednesday morning, when they came in to work, the job had terminated with an error, and then only had up to 6 February cleanly restored.

Moral: the cloud is vulnerable in the ways that people concerned with the provision of computing services have known about for a long time. This is not the first time this has happened to me (indeed, the third time I have lost amounts of mail in five years). There are obvious ways to avoid specific problems, but there is mostly neither time nor resources to implement and manage all those solutions perfectly all the time. In this case, there were (at least) two failures, and it is clearly impractical for the faculty computing services to check continuously whether they can effectively restore data through such two failures, as well as all the other possible failures that could occur. This is a resource-intensive on-demand function and it is combinatorially impossible to check regularly the execution of all such functions in even a moderately complex system such as e-mail backup.

When someone comes up with easy ways to solve any digital-computational vulnerabilities, say to GPS interference, that is less than half the tale. The rest of the tale concerns whether those solutions are implemented, and also continuously and effectively maintained.

There is a lot of superb computer science behind this nowadays. Versions of Leslie Lamport’s Paxos algorithms are enabling Google’s servers to provide us with our daily informational bread (Paxos logically serialises distributed database transactions).

Most journalists and digital-services marketing people have not heard of, let alone understand, the combinatorial impossibility of checking and maintaining all your on-demand functions, or even routinely how the various Paxos variants work and three-phase commit doesn’t. To find out what is possible and what is not, in other words, you still have to talk to computer scientists with authoritative knowledge. Such as Martyn and his GPS-vulnerability team from the Royal Academy of Engineering. And be wary of what is said in thoughtful articles about “cloud computing” in news media unless it comes from such people.

What actually happened to me? I don’t know. The “stuck delete key” hypothesis seems to me to be implausible (it has worked fine since). And a software glitch in my mail client alone would not explain why the windowing system pull-down menus failed to operate as expected. I am not unfamiliar with forensic analysis of this sort (indeed we do it for major accidents) but this is not the first time an explanation has eluded me and I doubt it will be the last.

Not mine this time (the one I wrote in 1997 is still being referenced, but is out of date because the German degree system has changed) but the OECD’s from October 2011, based on 2009 data, which I have just discovered. The Washington Post published in September 2011 a startling graphic, accompanying an article on the report to which was linked in an essay today by Nicolas Kristoff of the NYT. (Kristoff is a member of my college. In his journalistic wanderings around some of the poorest, most disadvantaged parts of the world, he sometimes seems to me like a modern Wilfred Thesiger, a former member.)

I should note, first, in reference to the Washington Post article that the US term “college” refers to all higher-education which leads to a qualification called a degree. This includes “community colleges”, tax-supported institutions which provide the equivalent of the first two years of a four-year university education and which grant degrees called “associate degrees” to successful students, as well as universities, which may be four-year or six-year institutions, as for example the California State University system is, granting Bachelor’s and Master’s degrees, or “research universities” such as the University of California which also grant Ph.D. degrees.

I recall British Prime Minister Blair saying in 1997 (do I?) that the Labor government intended to push degree achievement rates up to 35% of the population, up from the 15-18% or so which it was when I graduated in 1973. I didn’t realise until I looked at the WP graphic, based on the 2009 data, that this had been achieved. I herald it as a major national accomplishment.

(I get the figure of 15-18% as follows. This 2000 report by David Greenaway and Michelle Haynes says that about 400,000 young people were in tertiary education then. If one takes the average lifetime, a little under 80 years, considers that 3 years is a twenty-fifth of that, and that the population of Britain is about 60 million, one would expect 2.4 million people of university-visiting age. 400,000 is thus one in six, about 17%. I should perhaps mention that Laura Spence, who was rejected by “Oxford” but given a scholarship at Harvard, had in fact applied to my college. Not the greatest marketing moment in history).

Similarly, I had, until today, oft quoted the rate of young people in the US entering higher education as a sign of what I thought was desirable, and used the figure of 55% of school leavers. I doubt if this has changed significantly. But I am disturbed to find out that that apparently only 41%, about three-quarters, complete to some sort of degree. Considering that includes associate degrees, which are only two-year courses of study, that does not bode well for the US, if you believe as I do that the more people learning skills in a short time which they otherwise would not have, then the greater the productivity of their society, in the richness of hobbies and other pursuits in life and not just in stuff measured in standard economic measures.

I am intrigued by the Box on p18 of the OECD report entitled “Germany rethinks its assumptions about education and social equity”. Yes, indeed! People here were quite convinced about the “quality” of the education system, despite the obvious inequities and inadequacies apparent to those of us with wider experience, until the PISA reports on comparative achievement in secondary education started appearing from 2000 on, which showed German school achievement in a poor light compared with Germany’s economic peers. Then it couldn’t be ignored any more, and it wasn’t.

PISA was to do with secondary education. I am still somewhat disturbed by the relatively poor showing of Germany in tertiary education, at 26%. Some comments on that, some of which I have made before.

We currently have huge building projects going on around our Bielefeld University campus, which is itself huge (put “Bielefeld University, Universitätsstrasse 25, Bielefeld, Germany” into Google Maps). The main university building, in which almost everything goes on, is some third of a kilometer long, as you can see. Two new campuses are being constructed, one adjacent to the old building on a parking lot just to the north of the main building, between the two branches of what is labelled “Universitätsstrasse, some two hundred meters long and the better part of a hundred meters wide, and one “over the road”, almost a kilometer away, in (Google Maps again) “Lange Lage, Bielefeld, Germany”, which is also large, and will house the University of Applied Sciences (what the Brits used to call a “Polytechnic” and Germans a “Fachhochschule”), a teaching university which does not grant research degrees, and which is now largely scattered in old and often unsuitable buildings around town. This all amounts to a huge public works (which Google Street View does not yet show). And, if the above figure is to be believed, this will only be usable by a quarter of the young adults in the city and surrouding areas.

Do we have a town-and-gown problem? Less so than we did, I think, but more so than we might. The university does some outreach, including a science fair each year called Geniale (some pictures of GENIALE 2011 – the German for “pictures” is “Bilder”), spread over selected spots in the Old Town. But why aren’t most of the young people in this area passing through some part of this enormous spreading campus to take part in something? After all, they and their parents pay the taxes that create all these large buildings and pay their occupants. Future auto mechanics and hairdressers could surely benefit personally from participating in a course on 1960′s popular music, couldn’t they? Germany has no equivalents to Brian Patten, Roger McGough, Adrian Henri or Carol Ann Duffy, but we have plenty of slam poetry (link only in German, unfortunately), indeed a local slam poet who has turned into a valued writer and raconteur, Mischa-Sarim Verollet (also only German). Here is the announcement for the next one in April 2012.

Such educational offerings are available through the Volkshochschule Bielefeld, the Community Further Education Center, but this is largely less formal – courses are not assessed, the qualifications of course-offerers fulfil no standards (either experiential or formal), one doesn’t obtain a transcript of courses completed, and, importantly, it does not constitute the kind of accomplishment which a prospective employer expects to see on an applicant’s résumé. I am thinking that all these things should happen. I am also thinking about the impoverished financing of the Volkshochschule compared with the heroic building works around the university campus.

I cannot see that expensive tertiary education can thrive unless it includes way more than the elite. We are well past the days when people said “well, that’s for them rich and clever kids” and turned their backs. Nowadays, people say “I pay taxes too; why can’t I come in here?” and I think that question is very well founded. Especially when the expenditure is so massively visible, as it is in Bielefeld.

German university education has changed, though, massively in the last decade. The previous system has been more or less junked, and every university now offers Bachelor’s and Master’s degrees, instead of the old Vordiplom/Diplom, which were not recognised outside Germany for what they were (a Vordiplom was like a US associate degree, and a Diplom like a Master’s, but with nothing in between). It is astonishing how everyone just threw the old tradition away in the early 2000′s and went with what, for most here, was a completely foreign system with which they had little or no experience. I did find out why from a colleague in Sociology, though. They had over a 90% drop-out rate in their Diplom course. And this in one of the most well-reputed Sociology faculties in the country that invented it.

I think student contact with the rest of Europe was also slowly bringing a new perspective. German university students were finding themselves relatively immobile compared with their peers in other European countries, because the organisation of their degrees did not easily translate. For example, in the late 1990′s, students studying for degrees in my faculty returning from studying abroad for a year in the ERASMUS program still had to take an oral degree examination in the studies they had completed abroad to have it count for our degree, even though they had already been assessed by the foreign institution for that work and the EU ERASMUS agreement requires that we honor that assessment. To those who came to me, I asked for the transcript, or equivalent document showing successful completion, asked them to tell me about what interested them in the work, and passed them. In other words, the exam was purely formal, and the result identical to what they had already achieved. That is the best way I could see to fulfil the EU requirement, which our internal faculty procedures at that time still contradicted.

Besides that, successful graduates (the Sociologists’ 10%; our proportion in Informatics was much, much higher!) were leaving tertiary education with a degree equivalent to a Master’s at the age of 26-28 (and some even older), whereas their British and US peers were obtained such qualifications at the ages of 22-24. People on the ERASMUS exchange were noticing they were somewhat older than their local peers, and those starting Ph.D. programs in other countries noticed it even more.

Now, we have Bachelor’s and Master’s degrees, credit points for each course, and credit points are transferable between all European tertiary-educational establishments.

I cannot necessarily say that the quality of education has improved, however. With the more extensive evaluation requirements (per course, now), much of this is being farmed out to tutors and other helpers, and the quality of that education and assessment does not seem to be monitored as I feel it should be. I monitor the courses in my group, which are all based on lab work, or seminars which consist largely of student contributions with commentary from the lecturer, and my group has considerable continuity in our student tutors, who were picked for (or, better said, who picked themselves by) their enthusiasm and capabilities. But some of our larger courses appear to have problems (one of my bright people, who has coauthored an important chapter in our system safety text, is on his third attempt at one of the required practical courses, for what appear to me to be spurious reasons).

The throughput has, however, improved. One reason in the past was the introduction of modest fees, some few hundred euros per semester. Suddenly, all our 6-year and 7-year students (of which we had plenty) wanted to finish – and most did. And the fee money was directly given to the Faculty, in which a largely student committee, which did include the Dean, decided what to do with the money to finance improved teaching. More tutors for some courses. Lab equipment – my lab was built with this money. The faculty also hired a highly motivated and very successful lecturer whose courses are loved by students and who does lots of lab work, indeed he uses the lab which we built.

The other reason is that students in our Bachelor’s and Master’s programs are spending much of the day in courses, and most of the rest of their time doing the homework. Their time is filled with study-related work. This is very different from ten years ago. But I think it is a benefit, more on a par with what their peers do in other countries with a higher percentage of college graduates.