One of the reasons I’m writing the Nature of Internet Standards series is to give policymakers tools that help them to understand and evaluate open standards efforts. A big part of that is understanding whether a specification (and the effort that led to it) is really a reflection of open deliberation, consensus processes, appropriate governance, and various other hallmarks of open standards.

Evaluating these factors requires a fair amount of context and nuance. What’s the most 2025 way to avoid that work? Using AI, of course.

Some colleagues were using AI to ask similar questions, and that inspired me to wonder whether it would be a useful option for policymakers who want to evaluate the openness of Internet standards. Let’s find out.

The Privacy Sandbox

Our first experiment is an effort that I’m fairly familiar with, having engaged with it both professionally and academically – Google’s Privacy Sandbox. How will AI portray this longstanding and very politicised effort?

Prompt

I started by asking ChatGPT’s “deep research” mode:

Is the Privacy Sandbox developed in an open standards process? Consider the venue(s) it’s developed in, their governance, openness, and conformance to other widely recognised signifiers of open standards processes, breadth of implementation, and adoption.

It asked for a few clarifications:

To give you the most accurate and up-to-date analysis, could you clarify a few things:

1. Are you primarily interested in the Privacy Sandbox on the web (i.e., as developed by Chrome and the W3C), or are you also considering the Android Privacy Sandbox?
2. Do you want a comparison to established open standards processes (e.g., W3C, IETF), or just an evaluation of the Privacy Sandbox initiatives on their own merits?
3. Are you looking for an overview of industry support (browsers, advertisers, publishers, etc.) and real-world implementation as part of your evaluation of adoption?

This will help me tailor the research more precisely.

To which I replied:

Just on the Web. A comparison to open standards processes would be good. I don’t need an overview of support and implementation, but the results should take these factors into account.

Response

After churning through data (and pushing the boundaries of copyright) for a couple of minutes, ChatGPT came up with this voluminous report. However, the heart of its response is concise and fairly damning:

In summary, the Privacy Sandbox has been developed in semi-open forums (W3C community groups, public repos, Google blogs) and invites outside input, but it is not a widely supported multi-vendor standard yet. Only Google’s Chrome (and related browsers) implements the APIs, and other browser vendors have developed their own approaches. The process resembles an open-source Google project more than a classic consensus-driven SDO. As Mozilla and others have noted, standards bodies like the W3C and IETF are the “ideal forum” to vet such proposals with full stakeholder oversight. At present, Privacy Sandbox’s governance does not fully align with those open standards processes – it provides considerable transparency and documentation, but its development and adoption remain largely controlled by Google.

That’s not far off. W3C Community Groups aren’t standards processes; in fact, they’re little more than mailing lists where the proponents get to set almost all of the rules. ChatGPT could have made this more clear, but instead uses lots of hedging phrases like “more than” and “fully align”.

Since the full report mentioned the chair of the Business Group that also discussed the Privacy Sandbox, Wendy Seltzer, I asked what she thought. Her response:

It’s fascinating to follow the sources here, because the Wikipedia article appears to have gotten its last substantive updates in 2024, and is mostly focused on older material ([which is] sometimes wrong). A human researcher would recognize that as not the best authority for recent news. A better human researcher would also ask “is this the question you want to ask, since the non-deprecation of third party cookies has changed the landscape around all these initiatives?”

That illuminates a broader issue of missing context. The answer starts from the premise that “Privacy Sandbox” is something that could be developed in an open standards process, rather than parsing the question, is PS more than the set of APIs and proposals that might fit into standards processes? How do open standards fit into the overall strategy? The GPT report can work for you, an expert in the field who can fill in those aspects, but it could be misleading to someone without the same background.

Some of the sentences quoted from other sources seem off without the context of those sources, e.g. “In January 2020 Google invited ad-tech companies to join the W3C’s Improving Web Advertising Business Group (IWABG) to discuss and give feedback on Privacy Sandbox ideas” reads as implying that Google was the only one doing the inviting.

QUIC

Next, let’s try another effort that I’m familiar with: QUIC, the effort to define a new, performance and security-focused transport protocol. I chaired the QUIC Working Group (along with Lars Eggert) from its inception until it shipped its core deliverables.

Prompt

I asked ChatGPT’s “deep research” mode a similar question for QUIC:

Is QUIC developed in an open standards process? Consider the venue(s) it’s developed in, their governance, openness, and conformance to other widely recognised signifiers of open standards processes, breadth of implementation, and adoption.

and once again, it asked for a similar clarification:

To best address your question about QUIC, could you please clarify what aspect you’re most interested in?

• Are you focused more on the technical development and governance process (e.g., how open the standardization process is)?
• Or are you more interested in its current implementation and adoption across the industry?
• Should I compare it to other protocols or standards in terms of openness and adoption?

Let me know so I can tailor the research accordingly.

I responded:

Focus on how open the standardisation process was, but consider implementation and adoption in your answer. A comparison with other relevant standards would be helpful.

Response

Here’s the full report. The summary is positively glowing:

In conclusion, QUIC’s standardization has been as open and transparent as any recent IETF protocol. It was developed in the open IETF process, adhering to OpenStand and WTO/TBT ideals of consensus, transparency, and voluntary adoption. Its final RFCs affirm community consensus, and the IETF infrastructure provided public visibility into every stage. This openness is mirrored by QUIC’s broad support: independent implementers from Google, Mozilla, Apple, Cloudflare, Facebook, and beyond have all deployed QUIC. The resulting ecosystem – with multiple browsers, CDNs, servers and apps supporting QUIC – reinforces that it is a truly open standard, not controlled by any single vendor. By contrast, older protocols like HTTP/2 and TCP underwent similar open processes (IETF consensus, public RFCs) and likewise enjoy ubiquitous implementation. Together, the history and adoption of QUIC demonstrate an effective, open standardization process in line with global best practices.

First of all, I love the reference to the “WTO/TBT ideals of consensus, transparency, and voluntary adoption.”

But is it accurate? Sort of. The facts here are correct: those companies did participate in the effort, implemented it, and deployed it. The process was open (as can be seen in the minutes and other materials).

I don’t know, however, how it justifies its statements about the quality of consensus – as I’ve written before, consensus can go wrong in a variety of ways. Others have characterised QUIC’s development has with Google firmly in the “driving seat.” Having been there for the whole standards process, I disagree with that assessment, but I’d at least expect academic work like that to be acknowledged.

Is AI Useful for Assessing Open Standards?

I can’t count the number of times that I’ve seen policymakers, journalists, and community members refer to the Privacy Sandbox as “at the W3C” or “being standardised at the W3C.” Given that extremely low bar, ChatGPT’s summary is an improvement. Likewise, I largely agree with its assessment of QUIC, at a high level.

What’s lacking here, however, is any kind of nuance. I can’t escape the feeling that it latches onto a few narratives that appear in source materials and augments them into well-worn clichés, like we see for QUIC. The IETF has a great reputation in many sources, so that gets amplified, but there’s a lack of any critical thought.

That’s not surprising: AI can’t think. If it could, it might wonder about the criteria we’re using for “open standards” here – are those WTO/TBT ideals still relevant, and are they adequately described? Are the processes actually used in working groups lining up with the rhetoric of openness – and how would you find out if they didn’t? And, how much should all of that count if the result isn’t proven by market adoption?

In a nutshell: if you must use AI to assess the openness of a standard, only use it for the first pass, check all of the references, and then roll up your sleeves and start talking to people to get the real story.

The IRS has now published the vast majority of Direct File’s code on GitHub as open-source software. As a work of the U.S. government, Direct File is in the public domain. And now everyone can check it out.

Releasing Direct File’s source code demonstrates that the IRS is fulfilling its obligations under the SHARE IT Act (three weeks ahead of schedule!). Now that Direct File has paved the way, I hope that more of the IRS’s code, paid for with taxpayer dollars, will soon be available to all of us.

Open sourcing Direct File has long been planned, and even longer desired. Explaining last May why open source is particularly important for Direct File, the team wrote:

The IRS could take further steps to build public trust and enable independent assessment of its work. The Direct File product team was given the mandate to develop software that ensures every taxpayer receives the full benefit of any tax provisions for which they are eligible. Releasing components of Direct File as open-source software would enable the team to demonstrate this commitment.

Establishing trust with taxpayers was core to our approach for designing and building Direct File. By creating the most accurate option for filing, by making taxes accessible to all, by keeping taxpayer data secure, and now, by publicly sharing Direct File’s code, the Direct File team showed our dedication to earning taxpayers’ trust.

Please note: As of two weeks ago, I no longer work at the IRS. I am writing solely in my personal capacity.

Brittleness and resiliency. Read here.

Before I went to work for Audible (five years ago now — time flies!) I had a bias about engineers that worked for large corporations. I assumed that they weren’t as good as indies and engineers at small companies, or else they’d actually be indies or work at small shops like Omni.

Obviously I knew there had to be exceptions, particularly at Apple, or else we wouldn’t have had great things like AppKit and UIKit and everything else we’ve built on over these years. But the bias persisted.

* * *

Before Audible, the largest company I’d ever worked at (Newsgator) had just over 100 people. When I worked at Omni it had roughly half that number.

I’ve spent half my career working at even smaller companies, with just me and Sheila (Ranchero Software) or at places with three people (Q Branch) or like six people (UserLand Software).

And of course I was arrogant enough to think that I was better — much better — than any corporate engineer. While a corporate engineer might own some small part of an app or framework — or just a single button, as the (lame) joke went back in the day — I was shipping entire apps on my own or with a very small team. Popular, valuable, newsworthy apps that people loved.

And I wasn’t the only one: think of Flying Meat, Rogue Amoeba, Bare Bones, Red Sweater, The Iconfactory and many more.

* * *

And so I learned very quickly when I started at Audible that I was very wrong. I was impressed, and grew more impressed as time went on, by my fellow engineers’ rigor, talent, professionalism, care, and, especially, ability to work with other people toward common goals.

While I’m the die-hard introvert who just wants to go into a room and sit in front of a Mac and write some code and get things done, I learned that my co-workers — even if they, like me, kinda just wanted to sit and write code — were great at app development as a team sport. I was impressed with how they wanted to grow and did grow — always leveling-up their individual skills and their ability to work on a team and across teams.

And what a team it was! It’s not a new observation, but the indies I mentioned above, and the ones I didn’t, tend to be white men born in the United States — the people who could most afford to fail, in other words, because for them (for me, absolutely) there’s always another opportunity.

My team didn’t look like that — it was quite a contrast with my previous experience. Many more women, people of color, people born outside the United States. (But note that there’s always more progress to be made!)

The engineers on my team could write apps as well, if not better in many cases, than the indies I know. And the ones who aren’t quite there yet — well, just give them a little more time. They’ve all given me reason to believe in them.

I regret my bias about engineers working in corporate environments, and I’m so glad I learned the truth almost from day one on starting at Audible.

* * *

For a couple years I did a lot of hiring — a lot of interviews — at Audible. And I noticed something: there was a strong correlation between being hirable and having worked with other people.

The folks who’d worked largely by themselves, or on just one small team, weren’t as good candidates as the folks who’d worked with more people. This, of course, went against my original bias that indies are the best engineers — but by then I knew that a candidate who’d worked with lots of other people had been exposed to more code, more dilemmas, more challenges (technical and human), and they were not just more ready to work on a larger team but more knowledgeable. Even their individual skills were greater.

Advice time: if you’re a newer engineer, find ways to work with other people. Not just because you’re more likely to get hired at a place like Audible — but because, no matter where you want to work, you’ll be better at it.

You can’t just sit alone in front of your computer all day and write code and expect to be a great engineer.

Lesson learned!

* * *

With retirement imminent — this is my last job, and June 6 is my last day (maybe I’ve buried the lede here) — I want to thank my team publicly for how they’ve made me a better engineer and, more importantly, a better person. From the bottom of my heart.

I learned more from them than I could ever have taught; I got the better part of this deal.

Thank you, team! So much. ❤️

Builder.ai let you build a website or an app without coding — but with AI! Allegedly.

Builder was the great hope of Artificial Intelligence for the UK. It scored $450 million in venture funding — mostly from Microsoft and the Qatar Investment Fund.

Customers had mixed experiences with Builder. A lot of positive online reviews turned out to be written by Builder employees. The company also put several logos on their website of companies that were never its customers. [FT, 2024, archive]

Anyway, Builder finally went broke yesterday, after years of interesting financial activities and a few minor accounting scandals, such as allegedly falsified sales figures and an auditor with conflicts of interest.  [FT, archive]

CEO and founder Sachin Dev Duggal had stepped down in February — though he kept the job title “Chief Wizard.” [FT, archive]

Here’s how Builder’s front page — still working for the moment — says the process works: [Builder.ai, archive]

1. Chat to our AI, Natasha
2. Get a fixed price and accurate timings
3. Meet your own dedicated expert
4. AI assembles your app features like a LEGO set
5. Features are customised by human specialists
6. Your app is ready

The most “AI” part of the whole thing is Natasha the keyword bot at the start of the process.

Steps 3 and 5 are where A Guy Inside does the actual work: [builder.ai, archive]

Natasha recommends the best-suited developer for your app project, who then customises your code on our virtual desktop. We also use facial recognition to check that the developer working on your code is the same one Natasha picked.

That’s not AI, that’s outsourcing. Builder.ai was just hiring your work out to A Guy Instead. But it was marketed to its venture funders as “AI.”

Builder claimed that the fabulous Natasha AI did most of the development planning. Builder also claimed that Microsoft wanted to use Natasha to sell customers business app development. [press release, 2024; TFN, 2023]

But there’s no evidence that “Natasha” wasn’t just a human doing all the tricky bits. Because Builder had past form on using A Guy Instead.

Back when Builder was called Engineer.AI, they were caught in 2019 selling “AI” development with a bot called Natasha that turned out to be some guys in India who wrote the app for you. Engineer.AI told the Wall Street Journal at the time that the alleged AI was just “human-assisted.” Yeah, 100% human assisted. [WSJ, 2019, archive]

It’s not clear if anything’s left of Builder.ai. There’s no money and there’s no AI. But the real reward was the wild “off-site company meetings” along the way, where they splashed out the investor cash on a week in a five-star hotel in Vietnam for the employees. And Natasha the keyword bot. [FT, 2024, archive]

• Video version