Simon Willison’s Weblog

The realization hit me [when the GPT-3 paper came out] that an important property of the field flipped. In ~2011, progress in AI felt constrained primarily by algorithms. We needed better ideas, better modeling, better approaches to make further progress. If you offered me a 10X bigger computer, I’m not sure what I would have even used it for. GPT-3 paper showed that there was this thing that would just become better on a large variety of practical tasks, if you only trained a bigger one. Better algorithms become a bonus, not a necessity for progress in AGI. Possibly not forever and going forward, but at least locally and for the time being, in a very practical sense. Today, if you gave me a 10X bigger computer I would know exactly what to do with it, and then I’d ask for more.

— Andrej Karpathy # 30th May 2024, 7:27 am

But where the company once limited itself to gathering low-hanging fruit along the lines of “what time is the super bowl,” on Tuesday executives showcased generative AI tools that will someday plan an entire anniversary dinner, or cross-country-move, or trip abroad. A quarter-century into its existence, a company that once proudly served as an entry point to a web that it nourished with traffic and advertising revenue has begun to abstract that all away into an input for its large language models.

— Casey Newton # 15th May 2024, 10:23 pm

If we want LLMs to be less hype and more of a building block for creating useful everyday tools for people, AI companies’ shift away from scaling and AGI dreams to acting like regular product companies that focus on cost and customer value proposition is a welcome development.

— Arvind Narayanan # 15th May 2024, 4:25 pm

But unlike the phone system, we can’t separate an LLM’s data from its commands. One of the enormously powerful features of an LLM is that the data affects the code. We want the system to modify its operation when it gets new training data. We want it to change the way it works based on the commands we give it. The fact that LLMs self-modify based on their input data is a feature, not a bug. And it’s the very thing that enables prompt injection.

— Bruce Schneier # 15th May 2024, 1:34 pm

I’m no developer, but I got the AI part working in about an hour.

What took longer was the other stuff: identifying the problem, designing and building the UI, setting up the templating, routes and data architecture.

It reminded me that, in order to capitalise on the potential of AI technologies, we need to really invest in the other stuff too, especially data infrastructure.

It would be ironic, and a huge shame, if AI hype sucked all the investment out of those things.

— Tim Paul # 13th May 2024, 2:35 pm

Watching in real time as “slop” becomes a term of art. the way that “spam” became the term for unwanted emails, “slop” is going in the dictionary as the term for unwanted AI generated content

— @deepfates # 7th May 2024, 3:59 pm

I believe these things:
1. If you use generative tools to produce or modify your images, you have abandoned photointegrity.
2. That’s not always wrong. Sometimes you need an image of a space battle or a Triceratops family or whatever.
3. What is always wrong is using this stuff without disclosing it.

— Tim Bray # 4th May 2024, 4:26 pm

I used to have this singular focus on students writing code that they submit, and then I run test cases on the code to determine what their grade is. This is such a narrow view of what it means to be a software engineer, and I just felt that with generative AI, I’ve managed to overcome that restrictive view.

It’s an opportunity for me to assess their learning process of the whole software development [life cycle]—not just code. And I feel like my courses have opened up more and they’re much broader than they used to be. I can make students work on larger and more advanced projects.

— Daniel Zingaro # 3rd May 2024, 6:17 pm