Between 1996 and 2001 I had the great pleasure of working in one of the most dynamic teams at Nortel Networks. I was surrounded by a great applied mathematician, network engineer, product expert, economist, and botanist! Our small and very diverse team was located in Wollongong, Australia but serviced much of the pre-sales engineering support throughout Asia. My director would always emphasise to me- "make sure if you ever get into a position of leadership that you hire people with varying backgrounds that can add different perspectives and approaches to the same problem." I must say, getting into our 'WAR' room during each day was such a stimulating experience as we each vied for the whiteboard marker, and created simple but elaborate diagrams and action items.

Over the last few months I've been contemplating the Snowden revelations in light of my prior employment. Am I personally shocked by the visibility the NSA had? Not at all. And here's why.

Having worked on over 150 tender responses for numerous customers throughout Asia and even North America (while on secondment) I was made aware of a number of things: 

  1. All operators have incredible levels of detail about the very assets they own, i.e., their network infrastructure and how it is being utilised by their customers. Operators *have to* have this data for billing their customers and for fundamental teletraffic engineering and network dimensioning purposes, for managing the capacity and congestion of the network, and fault handling issues, among other issues like performance optimisation etc. While working for a vendor, I can categorically say I never ever received detailed information about customer call details.
  2. When we speak of teletraffic theory traditionally measured in Erlangs, most operators and service providers in the pre-data voice-only world were concerned about such measures as:  
  • the number of call attempts
  • call holding times (CHT)
  • mean holding time (MHT) 
  • busy hour of traffic (BH) 
  • originating traffic point
  • terminating traffic point
  • community of interest (CoI) 
  • quality of service (QoS) 
  • grade of service (GoS) 
  1. I have to add, that while we never received CDRs from operators, that in some cases we did not require them because our networking dimensioning estimations and traffic matrices for voice traffic between varying locations for instance was highly accurate as it was based on known consumer behavioural patterns. For network planning purposes it is enough to make a few high level assumptions with a few geodemographic facts like population figures, the number of households, the number of POTS lines, the penetration rate, how much of that traffic was going international, national or staying local (i.e. CoIs). Originating and terminating points for calls was an important issue in estimating total traffic and then figuring out how many lines or the number of trunks (e.g. E1s) required to carry that traffic.
  2. But what I do know is that even as far back as 1997, mobile operators were sorting customers into about 9 different categories of users in terms of profiling them over their usage behaviour and for estimating revenues for new services (e.g. SMS, MMS, WAP, mobile email, mobile gaming etc). Typically in my spreadsheet I would have mobile voice traffic and mobile data traffic estimates of each type of user.
  3. While vendors are only interested in aggregate data, operators need to have very detailed information. Which leads me to raising a point about MDFs in buildings. As the world became more data centric, what became increasingly important is what we call "network tails". Before Fibre-to-the-Curb (FTTC) was available, the network tails is what cost most companies and service providers a great deal of money- how to carry that data traffic to the last mile or last half mile. MDF stands for Main Distribution Frame and is typically a room located in the basement of a substantial building where there is substantial data traffic entering or exiting that company/building. Here is what an MDF looks like below.
Courtesy: Gregg Williams

Courtesy: Gregg Williams

It goes without saying that through Snowden's revelations we realise that governments are constantly watching us, and perhaps colluding to do so through their foreign acts in one another's jurisdictions. For example, the US used FISA as a reason for obtaining the CDRs, Australia will probably use another foreign intelligence act, and the UK another etc. By sharing this data they are then able to maintain they have abided by their countries laws and statutes.

When we were selling DMS switches worldwide, the switch would have to be compliant with local jurisdictions- e.g. in the USA it was CALEA at the time. If the technology was not compliant with the local jurisdiction then it could not be sold to customers in that country.

Now- as we have moved on with technology, here is where I make the claim that Snowden's revelations are just the tip of the iceberg. 

We are now carrying smart phones that come jam packed with sensors. If we believe that our data is NOT being profiled by at least our service provider or operator, then we are pretty much behind the times. Our service providers know when we switch on our phones and turn them off again, when we "pull" or "push" data to the handset (e.g. do a get request using Mobile Internet), how fast we are moving, and through location-based data make an inference on our mode of transport. If we push data to the Cloud using our mobile (e.g. to Facebook or Picasa or Flickr) then you can rest assured that your data could even be subpoenaed in the future for use as evidence based on the scene of a crime (time and date stamp of where you were when you took that photo). See my presentation and the panel I was hosting at the CEPS meeting on policing and human rights.

Courtesy: Katina Michael

Courtesy: Katina Michael

All of this is known explicitly as human activity monitoring. It is what predated the quantified self movement- QS is not new as some would have you believe. MG Michael and I guest edited a whole special issue on this theme in the Journal of Location Based Services. See our editorial on "The Social and Behavioral Implications of Location-Based Services" in issue 5 (2011) among the other critical papers in that issue.

It is really important that we acknowledge that the NSA, and other intelligence organisations can know a lot more about us than our call detail records- if they want to, on the consumer side they can know a lot more about our every touchpoint and interaction with our handset (and subsequently the Cloud), and on the business side, not only recognise  where our data traffic is heading and where it is coming from but what is in the content itself. A new breed of forensic scientists are emerging and they are not specialists in DNA analysis, but specialist in the digital realm.