AI Safety Paradox

Under several reasonable assumptions, super intelligence will actually help defenders in attacker-defender asymmetries that arise in biological or cyber warfare.

As the marginal cost of intelligence goes way down, many more attack vectors can be found via red-teaming and systems can get hardened or inoculated until all relevant attack vectors are covered. While this used to be impossible for most complex systems, super intelligence will make this much more viable.

Assumptions:

The marginal cost of intelligence is going down. 

There is a large but finite number of attack vectors N_d for a given system to be defended against.

Attackers have a budget B_a and defenders have a budget B_d.

There is a cost c to defend or exploit an attack vector and it’s much smaller than the total budget on each side c << B_a and B_d so theoretically many attacks can be found.

This cost is larger driven by the cost of intelligence.

Initial Intuition:

The cost of one attack is much smaller than the budget on each side. Attackers only have to be right once, whereas defenders have to be right c * N_d.

Hence, many people currently assume the attacker-defender asymmetry exists and just gets worse with more intelligence becoming cheaper.

Paradox:

What will likely happen instead is the following. If you divide the total defender budget by c, you get the number of currently available threat vectors you can pro-actively defend against: N =  B_d/c.

As the price of c -> 0, N approaches N_d. In other words, at that point, we can inoculate a system and defend against all potential attacks. The lower c, the more defenses can be built and the more we can harden a system.

Examples:

1. Biological Defense: An ASI could model every possible mutation of pathogens that can enter the human body and preemptively develop countermeasures—vaccines, treatments, or containment strategies—before an outbreak occurs.

2. Cybersecurity: In the realm of digital systems, an ASI could simulate all conceivable attack strategies against a network, identify bugs and vulnerabilities in real time, and patch them instantaneously.

Additional Notes:

• If attackers and defenders have access to the same open source intelligence or a similar level of reasoning, it will be even easier to predict potential attack vectors and then secure systems proactively. Open source makes it much more likely this paradox will play out benevolently.

• However, if attackers could reduce their costs massively and defenders cannot, this breaks down.

• In most cases, we can assume that B_d >B_a: the budget for defense is higher than for attack, eg. more money is available to save humanity from deadly viruses than is being spent on creating deadly viruses. 

• It is also fair to assume that the cost of an attack is lower than that of a defense. This doesn’t matter in the limit though if they both go down towards zero.

• If one assumed there are infinitely many attack vectors for a network or the human body, the paradox cannot play out well. But I’d argue that each has a finite limit, eg.

• While the maximum number of theoretical viruses reach into the millions, viable variations are limited to approximately 40,000 due to structural constraints, binding requirements, and evolutionary stability factors (according to a ydc compute agent analysis). https://you.com/search?q=How+many+viruses+could+there+be+that+can+attack+the+human+body+given+the+number+of+binding+sites+or...&cid=c1_0e4c322b-dd52-457b-bd3a-e35cc411b5e3

• In p(doom) sci-fi stories, fiction authors sometimes assume near magical capabilities, like a virus that can be dormant for years until it reaches all humans, stays completely undetected by everyone and then over night get triggered to kill everybody instantly. If such a near magical AI is available, I’d have used it to create a near magical vaccine against all such viruses given the cost is very low.  Near magical capabilities are generally highly unrealistic.

• This is somewhat connected to Jevon’s paradox (https://www.socher.org/thoughts/jevons-paradox-of-llms)

I look forward to well reasoned arguments for where this reasoning might be wrong.

Great Podcast Conversation

This was a fun conversation covering, AI, AGI, ASI, agents, search, productivity engines, investing and startups.

Other places to find it:

• YouTube: https://youtu.be/YK2Be7LcaK8

• Spotify: https://tinyurl.com/5n7dmfxk

• Apple Podcasts: https://tinyurl.com/yc88ndrb

Constructive Optimism

Constructive Optimism: A philosophical school of thought that recommends being optimistic about our ability to build incredible technology and systems. Even if the idea is too grand and not yet possible, we should try to make progress towards it with tangible, pragmatic milestones.

Examples: Artificial Super Intelligence. We may not get to it very soon, but aiming for it, defining it better, trying to build it will likely help us make incredible progress regardless. Same with becoming a multiplanetary species, world peace, defeating cancer, tripling health and life spans, etc.

I found the term being used before but not quite in this way. With increased complexity of our world, it is getting easier and easier to find flaws, issues, complaints and problems. It's getting harder and harder to find constructive solutions forward. This starts with story telling and continues all the way into engineering. I hope we can all nurture our inner constructive optimists more.

Jevons Paradox of Intelligence

Jevons Paradox of LLMs and Intelligence

When LLMs gets cheaper, we will not save costs but instead use more AI to automate additional workflows or increase accuracy.

Explanation:

GPT4 to 4o token costs dropped by almost 80% over the course of one year. Imagine how hard it was just 100 years ago to get an accurate answer on a complex chemistry problem. You would have had to travel and convince some famous researcher to talk to you. Now it’s at your fingertips.

Interestingly, one might think that with cheaper LLMs, we would see a lot of AI cost savings across organizations. However, instead, I think we will see Jevon's paradox happen here: When artificial intelligence gets cheaper, we will not save costs but instead just use more of it.

Automate more processes and hence possibly spend even more on the infrastructure.

Similar things have happened with engines and electricity.

We are entering the Age of AI

We are entering the Age of AI.

History does not repeat itself but it rhymes and this era combines aspects of the renaissance, enlightenment and the industrial revolution.

We have collectively never had so much access to knowledge. AI is making it more digestible with amazing answers and summarization. Despite its many problems, academia has never had as many bright minds doing research. AI will supercharge science and most other knowledge and research endeavors. There have never been so many artists exploring the human condition from so many angles. AI will enable anybody with an idea to share that idea in an artistic form. The speed of art will accelerate. We are overall going to be A LOT more productive and efficient in our work.

What will maybe be most distinctly different is that we will have a lot more people in total and percentage-wise with lives where most any of the basic human needs are fulfilled almost all the time. There's a certain kind of simplicity in struggle but complexity in success. You have to figure out what comes after that success. What will be our new collectively structured pursuit of wisdom, togetherness, belonging, grounding, meaning, or purpose?

Of course, we're also sitting on top of the advances of recent eras of steam, electricity, computer and internet and if you want to be pessimistic... maybe add some sprinkles of the middle ages.

As is always the case, the future (and any era) might already be here but not equally distributed. We're also for the first time acutely aware of all the places on earth that are struggling with war, famine and other catastrophes made by humans or nature. It creates even more pressure on our moral frameworks and how we define our in-group vs out-group, how we merge empathy with utilitarian effective altruism.

One thing is clear, if you were born from a random sample of all humans on earth, now is likely the most interesting time and your best shot to live a long life.

The Future Needs Better Marketing

The future needs better marketing these days.

While objective statistics like child mortality, literacy rates, deaths from Malaria etc have almost across the board improved many people are not excited about the future anymore or actually anxious.

Some want to hence step back into a past but some also want to constructively work on making the future better.

I think AI will obviously be the next step function in human productivity that will make our lives healthier, longer and without a lot of the boring repetitive types of work.

I am incredibly excited not just about making work more interesting and productive but also getting a much better grasp on science. Physics. Chemistry. Biology. When AI could connect to a more and more realistic simulation of eg a cell it can solve so many fundamental problems and make scientific fields into engineering disciplines. We really could build many ideas we've read about in science fiction.

Just one idea that has stuck with me: Carbon nanotubes with infused iron molecules and coated with proteins that only bind to brain cancer cells. They can lead to cell-level mechanical, nano surgery. So much more to come in this field in the coming years!

So incredibly exciting advances are coming. And generally when technology takes over a field or industry it means that access gets more equitable over time. Billionaires and middle class teenagers use the same phones! To think about the immediate future we can often look at wealthy people and then imagine how to enable anybody to have access to similar opportunities. Your own personal assistant. Personalized Healthcare teams. Personal tutors for your kids. Stem cell therapies. Helicopters/e-vtols, etc.

Now if there is indeed more and more abundance, automation of boring work and overall comfort coming there will also be a void or vacuum of meaning. Most people derive that meaning from their jobs and spend a lot of time on them.

Many have dropped religion, but haven't yet replaced it with another structured pursuit of wisdom, togetherness, belonging, grounding, meaning or purpose.

[This is a short welcome speech I gave at a recent Unconference we hosted here in the bay area.]

I'm excited about what this incredibly talented, intelligent, global and driven community will discuss today. I hope we get to learn from each other. Listen. Debate. And come out a little wiser at the end and then bring those lessons into the world.