Bookmarked How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt by Margaret-Anne Storey
I enjoyed this short posting by Margaret-Anne Storey, a CS professor. The effect of using generative tools can indeed lead to loss of overview, and uncertainty about the project I recognise. It creeps in very quickly, especially if I’ve started from something exploratory, as opposed to planned. A cognitive debt accrues because of wanting to move fast or move at all, at the cost of understanding one’s actions in enough detail. It hinders being able to make changes later.
It also makes me wonder something completely different. Partially because of examples I saw last week in Madrid of how BMW and Airbus had sped up some specific tasks orders of magnitude with AI:
If we see companies as slow AI, i.e. context blind algorithms working towards a narrowly defined singular goal (this is where the notion of AI turning all the material in the world including ourselves into paperclips comes from), what methods have we come up with to deal with cognitive debt in organisations? My intuitive response is reporting chains, KPIs, and middle management. Consultancy too, hiring an external actor to blame if needed. That suggests to me we actually didn’t, as so much of that is management-theater. Does any board of any company above a certain size actually know what is going on in their organisations? Understand what consequences changes may have? There’s a world of hurt out there caused by ‘reorganisations’ that all too often seem ritualistic more than rational when seen from the outside.
It may also be why companies easily embrace AI, despite e.g. warnings about cognitive debt. It looks the same as current practice, just with the promise of higher speed.
I saw this dynamic play out vividly in an entrepreneurship course I taught recently. .. one team hit a wall. They could no longer make even simple changes without breaking something unexpected. … no one on the team could explain why certain design decisions had been made or how different parts of the system were supposed to work together. … issue was that the theory of the system, their shared understanding, had fragmented or disappeared entirely. They had accumulated cognitive debt faster than technical debt, and it paralyzed them.
Margaret-Anne Storey