(or, put another way: I’m not sure a (non-legal, lay) definition of “conscious” is strict enough to be a useful frame for discussion!)

“I” is a strange loop. So, from that standpoint, yeah, there’s an argument: it could perceive itself in the (finite) regressive evaluation of tokens. I think the finer point is “are they aware” and “are they self-sustaining” to which I’d guess “no”

I was thinking the other day about the old-internet adage, “don’t read the comment section”, and how the internet gradually transformed most popular websites into infinitely-scrolling comment sections

the flip side of this is that, if you’re primarily able to program because of your day job, as is the case for folks early in their career, you’re strongly incentivized to pick projects with permissive licenses. you also have no/unknown leverage to argue to adopt projects with copyleft at that point

then it turned into popularity-as-job-insurance: it was easier (in theory) to get a job —or better yet, one closer to what you wanted— if you were the maintainer of a popular tool

honestly, in the past, it was that picking a permissive license and working in the open was a good way to build community around programming-as-public-art; it seemed to create the possibility for community members to get a job doing that art (in the django/rails/early node era of the web)

which discord group is this? (without being a member discord seems to refuse to display the thread)

I think a lot about Peter Naur’s “Programming as Theory Building” re: legacy code. A lot of the act of programming is really being able to model the system accurately along with the potential impact of changes to the system. (Legacy code is where that theory has been lost!)

I hope we’ll see package-lock-style prompt pinning come to clients— so that clients can alert users when the upstream server changes tool prompts (and hopefully block malicious tool prompts before they’re used)

So the de-skilling of new coders still makes me nervous– LLM-assisted coding is useful, but is shaped a lot like a Skinner box.

(Though this could change. If we get to a place where we put the LLM on local hardware and get it to produce acceptable output deterministically, that concern goes away)

We're able to not think about cache lines and electrical engineering because each of those interfaces aims to implement a deterministic virtual machine - they hide information.

For right now the lack of determinism in LLM behavior & output means the details of the output PL behavior peeks through.

From that angle treating human language as the HLL to LLM-generated code's LLL rhymes a lot with C/asm or asm/machine code.

It feels odd to generate human-oriented text just for machine consumption (but, as you know, it's one of CS's most cherished traditions, along with considering things harmful)

(you nerd-sniped me here–) code is a language intended for a human to precisely communicate intent with another human, one whose utterances can (incidentally) be automatically, deterministically interpreted by a certain class of virtual machine

Yeah - I guess I'm saying "it doesn't know what isn't written down, and what is written down isn't all there is to know about a system." It even omits writing down its own thought process for its changes.

(So "offloading" the mental model of a system to an agent is an easy-but-hazardous pattern)

(likewise! I hope you and the fam are doing well ❤️)

and when you combine an llm agent’s capacity for sweeping changes with this tendency, there’s a easy hazard of anthromorpizing the agent

the agent isn’t a domain expert; it can’t be. it reinvents its understanding of the code from scratch every session

put another way, the purpose of code review isn’t really to find bugs, it’s to communicate changes to the system to other team members; & sometimes those changes are large enough that the message is “okay, the person who committed this is now the domain expert on xyz”, …

I’m late to the party here, but ideally, the job of a team of programmers is to have a working mental model of the system under their care so that they can adapt it to changing needs; IME, llm agents are really good at typing but are (necessarily!) corrosive to this understanding