Simon Willison’s Weblog

Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it’s not even clear how prior LLMs learn anything at all.

— Andrej Karpathy # 2nd June 2024, 9:09 pm

Every year, some generation of engineers have to learn the concepts of “there is no silver bullet”, “use the right tech for the right problem”, “your are not google”, “rewriting a codebase every 2 years is not a good business decision”, “things cost money”.

— sametmax # 23rd June 2023, 11:59 pm

Back then [in 2012], no one was thinking about AI. You just keep uploading your images [to Adobe Stock] and you get your residuals every month and life goes on — then all of a sudden, you find out that they trained their AI on your images and on everybody’s images that they don’t own. And they’re calling it ‘ethical’ AI.

— Eric Urquhart # 22nd June 2023, 11:13 am

Cellphones are the worst thing that’s ever happened to movies. It’s awful. [...] I think you could talk to a hundred storytellers and they would all tell you the same thing. It’s so hard to manufacture drama when everybody can get a hold of everybody all the time. It’s just not as fun as in the old days when the phone would ring and you didn’t know who was calling.

— Steven Soderbergh # 12th June 2023, 6:13 pm

If you give feedback that isn’t constructive your feedback is worthless. I know that sounds harsh but it is. If you give unconstructive feedback you might as well not be saying anything. If you just look at something and go “That’s stupid” or “I don’t like that”—that’s worthless feedback, nobody can do anything with that. They’re not going to start throwing darts against the wall until you say “Oh OK, I like that”. You have to say something more.

— Timothy Cain # 5th June 2023, 4:58 pm

There was an exchange on Twitter a while back where someone said, ‘What is artificial intelligence?’ And someone else said, ‘A poor choice of words in 1954’. And, you know, they’re right. I think that if we had chosen a different phrase for it, back in the ’50s, we might have avoided a lot of the confusion that we’re having now.

— Ted Chiang # 4th June 2023, 2:59 pm

He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation.

[UPDATE: This turned out to be a “thought experiment” intentionally designed to illustrate how these things could go wrong.]

— Highlights from the RAeS Future Combat Air & Space Capabilities Summit # 1st June 2023, 11:07 pm

The general idea of an “Islands” architecture is deceptively simple: render HTML pages on the server, and inject placeholders or slots around highly dynamic regions. These placeholders/slots contain the server-rendered HTML output from their corresponding widget. They denote regions that can then be “hydrated” on the client into small self-contained widgets, reusing their server-rendered initial HTML.

— Jason Miller # 28th June 2022, 3:01 pm

Becoming a good engineer is about collecting experience. Each project, even small ones, is a chance to add new techniques and tools to your toolbox. Where this delivers even more value is when you can solve problems by pairing techniques learned on one project with tools learned working on another. It all adds up.

— Addy Osmani # 18th June 2022, 9:21 pm

In 2015, the men controlling 80% of Bitcoin mining stood on stage together at a conference. Three or four entities have run Bitcoin mining since then. The only thing preventing miner misbehaviour is wanting to avoid spooking the suckers — it’s completely trust-based. Bitcoin now uses a country’s worth of electricity for no actual reason. You could do the transactions on a 2007 iPhone.

— David Gerard # 28th June 2021, 5:32 pm

When I was a performance consultant I’d show up to random companies who wanted me to fix their computer performance issues. If they trusted me with a login to their production servers, I could help them a lot quicker. To get that trust I knew which tools looked but didn’t touch: Which were observability tools and which were experimental tools. “I’ll start with observability tools only” is something I’d say at the start of every engagement.

— Brendan Gregg # 8th June 2021, 7:33 pm

I’m pretty convinced that the biggest single contributor to improved software in my lifetime wasn’t object-orientation or higher-level languages or functional programming or strong typing or MVC or anything else: It was the rise of testing culture.

— Tim Bray # 1st June 2021, 2:35 pm

Data Science is a lot like Harry Potter, except there’s no magic, it’s just math, and instead of a sorting hat you just sort the data with a Python script.

— GPT-3, shepherded by Max Woolf # 29th June 2020, 4:45 am

Here’s a common piece of advice from people who create things: to make better things, make more things. Not only does it give you constant practice at making things, but it gives you more chances at lucking into making a good thing.

— Ned Batchelder # 28th June 2020, 2:29 pm

If you have to choose between engineering and ML, choose engineering. It’s easier for great engineers to pick up ML knowledge, but it’s a lot harder for ML experts to become great engineers.

— Chip Huyen # 24th June 2020, 5:24 am

Without touching upon the question of who’s right and who’s wrong in the specific case of Basecamp’s Hey app, or the broader questions of what, if anything, ought to change in Apple’s App Store policies, an undeniable and important undercurrent to this story is that the business model policies of the App Store have resulted in a tremendous amount of resentment. This spans the entire gamut from one-person indies all the way up to the handful of large corporations that can be considered Apple’s peers or near-peers.

— John Gruber # 20th June 2020, 5:48 am

There’s a spectrum on YouTube between the calm section — the Walter Cronkite, Carl Sagan part — and Crazytown, where the extreme stuff is. If I’m YouTube and I want you to watch more, I’m always going to steer you toward Crazytown.

— Tristan Harris, former design ethicist at Google # 9th June 2019, 6:22 pm

Raccoons don’t think ahead very much, so raccoons don’t have very good impulse control. I don’t think the raccoon realized when it started climbing what it was in for.

— Suzanne MacDonald, raccoon behavior expert # 15th June 2018, 6:06 pm

One of the ways the internet has changed around us over the years is the blog-o-sphere of MetaFilter’s early years has all but disappeared, and so has the kind of link-sharing culture that went with it.

— Josh Millard # 14th June 2018, 2:01 pm

Open Source gives engineers the power to collaborate across legal entities (companies) without involving bizdev. The benefits of this workaround are extraordinary and underappreciated.

— Yehuda Katz # 6th June 2018, 9:52 pm

At Harvard we’ve built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their JupyterHub environment. Our most ambitious so far has been setting up each student in the course with a p2.xlarge machine with cuda and TensorFlow so they could do deep learning work for their final projects. We supported 15 courses last year, and got deployment time for an implementation down to only 2-3 hours.

— Chris Rogers # 5th June 2018, 7:37 pm

Half of the time when companies say they need “AI” what they really need is a SELECT clause with GROUP BY.

— Mat Velloso # 1st June 2018, 2:35 pm

We can deploy new versions of our software, make database schema changes, or even rotate our primary database server, all without failing to respond to a single request. We can accomplish this because we gave ourselves the ability suspend our traffic, which gives us a window of a few seconds to make some changes before letting the requests through. To make this happen, we built a custom HTTP server and application dispatching infrastructure around Python’s Tornado and Redis.

— Dan Manges, Braintree # 30th June 2011, 9:27 pm

I’m renaming the book to “Dive Into HTML 5” for better SEO. This is not a joke. The book is the #5 search result for “HTML5” (no space) but #13 for “HTML 5” (with a space). I get 514 visitors a day searching Google for “HTML5” but only 53 visitors a day searching for “HTML 5”.

— Mark Pilgrim # 8th June 2010, 8:48 pm

Software engineers today are about 200-400% more productive than software engineers were 10 years ago because of open source software, better programming tools, common libraries, easier access to information, better education, and other factors. This means that one engineer today can do what 3-5 people did in 1999!

— Auren Hoffman # 24th June 2009, 11 am

You can buy an iPod nano on Apple, Best Buy, etc. for about $149. Amazon sells it for $134. That’s probably cost price. It turns out that Amazon can sell almost everything at cost price and still make a product because of volume. It’s all down to the Negative Operating Cycle. Amazon turns over its inventory every 20 days whereas Best Buy takes 74 days. Standard retail term payments take 45 days. So Best Buy is in debt between day 45 and day 74. Amazon, on the other hand, are sitting on cash between day 20 and day 45. In that time, they can invest that money. That’s where their profit comes from.

— Jared Spool, via Jeremy Keith # 22nd June 2009, 5:13 pm

And that is why, in 2009, when developing in Microsoft .NET 3.5 for ASP.NET MVC 1.0 on a Windows 7 system, you cannot include /com\d(\..*)?, /lpt\d(\..*)?, /con(\..*)?, /aux(\..*)?, /prn(\..*)?, or /nul(\..*)? in any of your routes.

— Benjamin Pollack # 12th June 2009, 11:48 pm

Let’s try to imagine what a Google Silverlight would have been. It would have been a fully open source product from Google, with a very liberal open source license (BSD or Apache). It would have all the technical specifications published openly. They would pledge to have the Silverlight VM interoperate with Javascript and HTML5. And a company like Zoho would have a ton of developers working on Google Silverlight based applications by now—as opposed to having exactly ZERO developers working on Microsoft Silverlight.

— Sridhar Vembu # 7th June 2009, 11:32 am

Bill Gates has pulled off one of the greatest hacks in technology and business history, by turning Microsoft’s success into a force for social responsibility. Imagine imposing a tax on every corporation in the developed world, collecting $100 per white-collar worker per year, and then directing one third of the proceeds to curing AIDS and malaria.

— Anil Dash # 26th June 2008, 5:17 pm

You may find that there are plenty of job listings where the job requirements are described as, “must be expert with Photoshop and Illustrator…” or something long those lines. Ignore those job listings; they’re placed by inept and sick companies looking for decorators, not designers.

— Andy Rutledge # 25th June 2008, 7:17 pm