it seems important to note that Davidson in this essay (the response piece, "A Nice Derangement of Epitaphs," to the 1985 Blackwell volume devoted to his work) is responding to the notion of "a language" that had become dominant in Chomskyan linguistics and in the many Chomsky-derived philosophies of language that were often used in opposition to Davidson, especially in the 1970s and 1980s, and as expressed in works like Chomsky's Knowledge of Language (which came out around the same time, based on lectures Chomsky had been delivering for several years). There are many philosophers (Wittgensteinians, as Davidson notes and in at least some ways was) and linguists who are not orthodox Chomskyans (such as James McCawley, whom Davidson approvingly quotes a page or two before) who do not believe in the notion of "a language" as such that Davidson is critiquing.  (this is a technical point, as they certainly do believe that there is a thing called "French" in practical enough terms for it to be the the thing a linguist specializes in, but as for what the boundaries and character of that thing called "French" are, they are far hazier and less easily specified than, say, the boundaries and character of R or C++).

and note the next sentence: "We must give up the idea of a clearly defined shared structure which language-users acquire and then apply to cases."

programming and formal languages are all about clearly defined structure, and this is one of the main things that proves so evanescent when one really starts to look at human language (and it is definitely the case that Chomsky for decades insisted that such structures were at least available for posit, and that in his latest version, the Minimalist Program, he has almost completely abandoned them). in fact, I believe it would be possible to show that what Davidson actually means here is something very close to the following: human languages are not systematic objects that one can learn and master in a discrete sense, in the way that one can and does learn and master programming and formal languages (and many other systematic and procedural skills). Though proving this would require some textual exegesis.