"Unable to move even a single step": Mechanism, policy and practice in the disciplinary smart city

Adrian Short · 24 November 2018

Beijing city government’s plan for implementing China’s “social credit” system provides a useful – and chilling – example of how infrastructures of social control work.

Bloomberg reports:

China’s plan to judge each of its 1.3 billion people based on their social behavior is moving a step closer to reality, with Beijing set to adopt a lifelong points program by 2021 that assigns personalized ratings for each resident.

The capital city will pool data from several departments to reward and punish some 22 million citizens based on their actions and reputations by the end of 2020, according to a plan posted on the Beijing municipal government’s website on Monday. Those with better so-called social credit will get “green channel” benefits while those who violate laws will find life more difficult.

The Beijing project will improve blacklist systems so that those deemed untrustworthy will be “unable to move even a single step,” according to the government’s plan.

We can think of it in terms of the mechanisms by which decisions are applied to citizens, the policies according to which decisions are made, and the practice of specific decisions applied to citizens. These form a hierarchy of social control infrastructure, with mechanisms providing a relatively stable base, relatively fluid policies providing the rules of the system, and the body of resulting decisions being the visible tip of the iceberg in citizens’ everyday life.

Separating mechanism and policy is a design principle in computer science. Systems should be designed so that the means by which decisions are implemented and the rules according to which decisions are made are loosely coupled. This makes it possible for the system’s owner to flexibly change the rules as required without having to alter the mechanisms by which decisions made according to the polices are implemented. Wikipedia uses the example of a key card system for doors in a building. The policies according to which cards will open which doors (and at which times) can be altered in the software without having to change the mechanism of the physical doors, locks and cards.

An obvious implementation of this system would be that when a person is no longer authorised to enter the building (ie a member of staff who has left) then their card is automatically disabled. This contrasts with a purely physical system of doors, locks and keys whereby physical keys would have to be retrieved from people who were no longer authorised to use them and where the theft or loss of keys could lead to the expense and inconvenience of changing the locks and issuing new keys to authorised people.

Mechanisms

In Beijing’s proposed system, the mechanisms include the way in which the state has required government and commercial systems such as airline and rail booking systems to require state authorisation before a ticket can be issued. In effect, this means that no-one can travel on public transport without the state’s explicit authorisation in every case. This particular mechanism is a man-in-the-middle attack on citizens’ mobility.

From the Bloomberg article, which bizarrely subheads this as “Ambitious Plan”:

Hangzhou rolled out its personal credit system earlier this year, rewarding “pro-social behaviors” such as volunteer work and blood donations while punishing those who violate traffic laws and charge under-the-table fees. By the end of May, people with bad credit in China have been blocked from booking more than 11 million flights and 4 million high-speed train trips, according to the National Development and Reform Commission.

Such mechanisms are presumably implemented either according to blacklists that the service providers hold or even by using live API calls to a government web service at the time of booking: the decision of the state to either grant or refuse permission for that specific journey at that point in time is then implemented by the provider’s system. An API-based system would be even more pernicious than a local blacklist system because it would give the state another surveillance mechanism by which every request for a travel ticket would be recorded: the state would know everyone’s intended movements, whether or not they were blacklisted, whether or not permission was granted in any specific case, and even whether or not the intended journey was ever made. In effect this is surveillance not just of people’s actions but of their intentions.

Policies

With the mechanisms in place for the state to, for example, prevent citizens buying travel tickets, we should consider the polices according to which decisions are made. China is an authoritarian state, so there is no particular restraint on the government to devise policies that are fair, reasonable or non-discriminatory. There is also nothing to stop the state implementing policies opaquely, whereby citizens are subject to the application of rules that are unclear, unreasonably complex, unspecific, or so broad in their potential scope that they can be applied arbitrarily.

A key principle of the rule of law is that laws should be sufficiently clear and precise such that someone subject to them should reasonably be able to discern in advance whether a course of action would be lawful or not.

Justice also requires that new laws are not applied retroactively, so that people who have acted in a way that is lawful at the time are not subsequently punished when the law is changed. China’s social credit system provides none of these safeguards. Rather, it implements a system by which fine-grained control of social behaviour can be exerted purely according to the changing whims of the state.

The social credit system creates a whole raft of extrajudicial punishments whereby state sanctions are applied without there necessarily being a clear charge, a fair trial, a system of appeal or even a defined end to the punishment. The sanctions will continue indefinitely unless and until the state deems according to the rules applied in the present that the citizen has redeemed themselves. Thus, a citizen who has received a parking ticket, a lawful and specific defined sanction in itself, may find themselves unable to buy a train ticket for a period of time, or even indefinitely, depending on the constantly shifting rules and the state’s view of the balance of their supposedly pro-social and anti-social conduct.

The result is an extrajudicial system of punishments that disintermediates the judiciary and the wider criminal justice system and puts the state’s tools to coerce and constrain directly into the hands of the government.

Practices

Everyday life under a social credit system such as China’s bears all the hallmarks of Bentham’s panopticon: the citizen is made to feel as if they are being observed at every moment, whether or not they are. The citizen correctly fears that sanction could be applied at any time, without warning or opportunity to appeal. The citizen becomes fearful and emotionally dependent on the reasonableness of the state’s rules and the fairness of their application. They have no ability to be able to ensure either, whether as a citizen individually or whether through collective political action. Effectively, the country is a prison and every citizen is a prisoner: condemned to live under the state’s sight and sword not for what they have done as individuals but for what they could do collectively if they genuinely had the freedom to act independently.

In Foucauldian terms this is a classic disciplinary system: play by the rules, assuming you can even discern what they are, and you will be left alone by the state or you may even benefit. Breach the rules and your ability to function in society becomes increasingly conditional until the point where you are literally “unable to move even a single step”.

Our politics and society

China is an authoritarian state, so it’s no surprise that it will use every available technology to maintain and extend its rule into the furthest reaches of its citizens’ lives. One lesson for those of us living in supposedly liberal democracies such as the UK is that we should resist when the state, in partnership with business, starts to construct the mechanisms that form the base of such a social control system. Systems that perform or permit surveillance of everyday activities are the foundation of a society that seeks to control them, whether through immediate “denial of service” interventions or through subsequent sanctions.

A foundation of free people living a free society is that their everyday activities are effectively unobserved and unrecorded: walking down the street, using public and private transport, buying and selling, reading, watching and listening, communicating, organising and assembling with others. Free people in free societies do not leave a data trail that the state and its business partners can use to analyse, influence and coerce citizens’ behaviour, individually or collectively. Free people in free societies do not delegate the power to do a thing or use a thing in everyday life to a remote authority, whether their decisions are issued via a network or by an algorithmic system embedded in the fabric of our environment.

We should require and demand analogue equivalent rights and freedoms: If we can read a printed book that we possess unobserved, unrecorded and without any authority’s ability not just right to prevent us from doing so, so should it be with the digital equivalent of that book. We should demand that public transport systems let us pay with cash, or better still, are free at the point of use. Having a right to the city fundamentally requires the freedom to move around it. No-one should be prevented from doing so by either state or corporate sanction or by financial hardship. In London we are seeing how the abolition of cash fares is leading to increasingly sophisticated surveillance and control systems like the “pay by face” idea that is the ultimate aim of the project being piloted at Cambridge Heath station next week.

Free people in free societies do not rely on the state to use its sight and sword judiciously. Rather, they constrain the state and its business partners such that their sight only extends so far and their sword only has limited reach. Free people in free societies appreciate that the vast majority of social space is unobserved and unregulated from above. Left to their own devices, free people will regulate themselves according to flexible and forgiving local custom and practice, with all the autonomy, negotiation and accommodation that requires.

UK citizens and those in other democracies have already ceded far too much ground towards the creation of social control systems such as China’s. Collectively, we have enthusiastically cheered the construction of an online panopticon where every microscopic activity is observed and recorded by multiple, usually opaque, entities. We have welcomed the migration of the majority of social, intellectual, commercial and political life to that system. In the name of further supposed efficiency and convenience we are extending the worst aspects of the internet – surveillance, opaque control, advertising and influencing – into the physical fabric of everyday life.

Against the “smart” life

So-called smart cities, smart homes, smart vehicles and other internet-of-things products are making our ability to act in every way subject to observation by and conditional on the permission of ever larger, ever more remote governments and businesses. The transition from individual ownership to use (ie ongoing rental) of even trivial everyday products and their integration into internet-connected product/service systems means we have to keep our money flowing towards the corporates while they extend their ability to observe and control our use of the things we used to own.

Who now keeps all of their data on their own computers rather than on a company’s “cloud”? Who wants BT’s phoneboxes to be replaced by Google’s InLink system that embeds three cameras in every kiosk and is designed to track pedestrian and vehicle movements while bombarding us with increasingly “contextual” (ie surveillance-driven) advertising? How long before our effective ability to own, maintain and use something as simple as a bike is squeezed out by state and corporate bike hires like China’s Ofo with their own disciplinary systems and complex, “connected” long-term hire bikes like those from VanMoof that track their (ie your) location and can only be repaired by the manufacturer? Who wants to live in a home where the landlord can remotely and automatically lock and unlock the doors at any time? Who wants to be chipped like an animal so that your boss can account for your location and activity down to the second? And who wants to live in a society where these kinds of things are widely practiced and considered good?

Take back control

Putting these surveillance and control mechanisms in place requires us to have enormous faith that they won’t be used in ways that further enhance the power of the powerful and further marginalise the powerless. But holding that faith requires us to ignore the experience of humanity’s entire history that shows that the use of power grows along with the ability to wield it. Once the mechanisms are in place, the policies and practices of social control are much easier to implement. As a matter of civic hygiene we need to stop building a prison around us while we hope and pray for the ongoing benevolence of the guards. We need to practice walking free while we are still able to move more than a single step.