The AI Strategist - 23.06.23

1. Hamburg Hackathon

Hamburg

I am writing this newsletter from sunny Hamburg where I am attending a Google-hosted Hackathon for CHAPTR.

The hot topic right now is agents. Most of the hackathon teams are using them in some capacity.

A real world example. Imagine you are running an AI Startup. You want to build a tool to help you identify relevant academic papers, Wikipedia, access information about the competition, what influential people are talking about, find out what is trending. Using agents can help with a few things:

1 - Connect to the web.

2 - Analyse a large volume of information.

3 - Save time, conduct research tasks for me.

4 - Ideate - come up with novel ideas around a problem.

5 - Suggest an action plan for an idea

The really interesting aspect of this for me is how we leverage this new technology while considering the process of intelligence (humans vs machines):

Layer 1 - Environment / perception - how we perceive the outside world. Humans have innate models (pre-installed) and world models. Humans are good at processing messy and incomplete data. For machines, data ingestion, cleaning and labelling is intensive. In this layer humans are more able to deal with incomplete data.

Layer 2 - Understanding - where low level sensory data gets represented into knowledge. It gets connected to prior data and abstracted into symbolic representation that the human mind can reason with. New data influences the human world model. Humans represent context and abstract lower level data to turn it into semantic objects (higher level) than we can manipulate. Machines do this terribly, they need hand-drawn maps of the world (called knowledge graphs). GPT4 is more powerful because of the integration of knowledge graphs. Hence more intelligent.

Layer 3 - Learning - machines do this incredibly well, much better than humans. 

Layer 4 - Abstraction / reasoning - once humans have all of these symbols, we can reason about them and evolve them inside our world model. Humans do this very well.

Why this matters - when organisations think about the best way to get value from agents, there are a few things to consider:

1. The only layer that machines do better than humans is layer 3. It is sensible to use agents to sift through vast amounts of information available on the web. This would take a human a long time and they would do a bad job.

2. As we begin to automate more tasks/workflows through LLM-driven agents we are not talking about deterministic processes (guaranteed input/guaranteed output) which is essentially traditional automation software. We are embarking on a new era where agents allow what we could refer to as ‘intelligent automation.’ Or perhaps better described as ‘not dumb automation.’ The critical point here is that you are putting quite a lot of faith in an LLM to ‘do and think’ for you without fully understanding how it does it. It is black box. It is up to organisations to decide how and why they want to benefit from this new opportunity. Successful Generative AI organisations will leverage layer 3 (saves time) but proceed cautiously on the emerging layer 2 and layer 4 properties LLM are displaying. This requires critical thinking around the resources you have at your disposal and how you get agents to work in partnership with your employees, not replacing it.

3. Personalities - you can give agents personalities such as ‘CTO’ and ‘Software Engineer.’ You can also ‘simulate’ conversations between agents and choose how much human involvement you want in crafting and shaping the output you need. So imagine if a human can accept, partially accept or reject an outcome. The fascinating aspect of this is how much improvement (in quality and diversity of thought) do these personalities give to a solution.

4. AI Safety - we are a long way off LLM-explainability but having an auditable system to show agent-agent conversations and/or human-agent will at least give organisations evidence of transparency of how they have used the agents to tease the outputs out of the LLM.
   

Systemic Safety

In the news - while following the tragic circumstances of the Titan submarine, a few quotes really stood out for me:

David Marquet, a retired US Navy captain, said OceanGate may have gone “a little too far.”

OceanGate 2019 blog post. "By definition, innovation is outside of an already accepted system. However, this does not mean that OceanGate does meet standards where they apply, but it does mean that innovation often falls outside of the existing industry paradigm."

Why this matters - many experts have been alluding to the fact that innovation is far ahead of regulation. It struck me that this is exactly how we should be thinking about Generative AI. We tend to forget that this can also lead to equally devastating outcomes.

Thinking about AI Safety in the context of systemic safety - something bad could happen in the real world. For example, a large language model produces racist content. We will call this a hazard.  

The risk from that hazard can be split into four factors: 

1. The probability of the hazard occurring. This increases if the foundational model was trained on racist content. 

2. The severity of the hazard. Racism comes in different forms: internalised, interpersonal, microagressions, institutional or structural.

3. Our vulnerability to the hazard. How humans react to this content.

4. The exposure to the hazard. How widely any generated content is shared.

Investments in AI Safety attempt to reduce some factor in this risk equation:

Being a whistleblower in big tech is not for the faint hearted. For the future of Generative AI, we need to trust companies to build environments where engineers, data scientists, commercial people, lawyers etc can have ethical conversations about outputs without fear. The systemic safety system breaks when people are scared to speak up, or worse, in OceanGate’s case they ‘removed’ the very person raising safety concerns.