One little underscore reveals overloading

1

One little underscore reveals a hacky design move: to support DKIM (a protocol that authenticates email senders to fight spoofing), domains store their public keys in DNS, in a CNAME record, using a fake hostname of the form

s1._domainkey.foo.com

Why the underscore? Presumably so that the special string _domainkey doesn’t clash with a real hostname, since hostnames are not allowed to start with one. But then the CNAME record is supposed to map hostnames, and so some DNS providers prohibit underscores in them. Oops!

This is a classic example of overloading: gory details here.

I wonder: is this kind of thing responsible for much of the complexity in these kinds of systems, which makes basic sys admin tasks often very challenging?

2

Hi.

The underscore conventions is a standardized practice and is actually quite popular:

[Scoped Interpretation of DNS Resource Records through “Underscored” Naming of Attribute Leaves]
(RFC 8552 - Scoped Interpretation of DNS Resource Records through "Underscored" Naming of Attribute Leaves)

It looks like you believe domain names are supposed to be restricted to host names, but that was never in its design. Layered conventions are at the heart of many – /very/ many – Internet standards enhancement efforts, over the decades.

btw, from your blog entry, you note that some DNS service providers do not support underscore. Indeed, there are many examples of software or operations that fail to conform to standards. That hardly demonstrates that there is a problem with the standard.

3

Hi Dave,

Welcome!

I see you yourself are the author of 8552, so you obviously know much more about this than me!

No, I was not suggesting that domain names have to be hostnames. What I was saying was that there are two distinct purposes or functions that DNS provides, which are distinct enough to merit being viewed (in my terminology) as two different concepts. One is name resolution for routing, and the other essentially uses DNS as a database for arbitrary properties.

My understanding (correct me if I’m wrong) is that hostnames used in routing should not have underscores in them, but names with underscores can be used for the second purpose, and in that case they’re not hostnames.

Now the concept design point: it seems to me that the DNS protocol doesn’t support the second purpose fully enough: hence problems with TXT lookups returning records that aren’t of interest, eg. So instead of supporting that purpose in its own right, for example in the manner I suggested, people started using other record types, such as the use of CNAME for DKIM. But that now creates some confusion, because CNAME is designed for name resolution and (unlike TXT eg) not database lookup. So understandably some DNS providers thought that CNAME labels should not contain underscores, and that leads to trouble.

Does that make sense?

Daniel

1 Like
4

Daniel,

Taking this as a technical forum, I’ll indulge in some quibbling, along with trying to deal with more serious substance…

The term ‘database’ has a generic, simple, non-technical use, but in more technical contexts, it tends to refer to a relatively rich set of functionality, including searching. So, for example, rather than specifying exact details for getting a specific record, it can search based on partial names or by attributes. I note this, because the DNS does not support such functionality, which is an aspect of the DNS that is often missed. The technology of the DNS is typically referred to as doing a ‘lookup’ rather than a search, since the DNS requires an exact match to a specific name. So, it’s best thought of as doing simple key/value processing. That’s a quibble.

Another quibble is that your reference to ‘routing’ is understandable but, really, the information returned for such queries is an address, not a route. (Since I always look for an excuse to cite this paper, thanks for that excuse: https://dl.acm.org/doi/10.1145/9760.9761.) That said, the MX record creates the global routing mechanism for email. sigh.

Perhaps more substantive is the distinction between what a mechanism does vs. how it is used. The DNS has a limited, fixed set of functions. Arguably, really, it only has one: Take a specific name and return some attributes associated with it. Period. It ‘understands’ no semantics to any of this. For example, it does not distinguish between attributes containing addresses vs those containing some other kind of information.

So the distinctions occur at a higher layer and vary on the /use/ of the DNS, not on the underlying structure or function of the DNS itself. Limiting the lookup string to what is called a hostname, is a feature of such a higher-level use. The DNS, itself, does not know about that limitation. And, again, it does not know about ‘routing’ – actually ‘addressing’ – use, versus other use.

I haven’t reviewed the specification details for where hostname syntax is required, versus allowing fully general DNS node names. Those details are for a use of the DNS, not as something in the DNS itself. (As I recall, DNS documentation contains the concept of a host name, but discussion about it is, I believe, non-normative. Maybe I’ve got that wrong.) Anyhow, you are certainly correct that a node name containing an underscore means the domain name is not a host name. That is, in fact, why underscore was chosen: no chance of colliding with a hostname.

Getting to your ‘concept design’ point. Limitations such as not distinguishing among a set of TXT records under the same name are indeed a hassle. The underscore hack is a way of eliminating or at least reducing the problem. Distinctive RR records would be cleaner, of course, but the field experience with the DNS is that getting end-to-end support for new RRs has had an extremely problematic history. After some decades with this reality, people starting just working around it. Ugly, but practical.

To clarify: the underscore convention permits defining a place for semantically-constrained attributes, such as in TXT records. Such TXT records will be there for that semantic and not some other. (It’s not unreasonable to simplify all this to: The underscore naming hack separates unrelated records of the same type, such as TXT RRs.)

The reference to CNAME in DKIM is interesting. Note that RFC 6376 does not mention CNAME. While use of CNAMEs with DKIM is extremely common – and noted in an Informational RFC – it’s not in the DKIM spec itself.

A quick scan of some online documentation about this does treat things as TXT vs. CNAME. For sysadmin’s purposes, that simplification is probably reasonable. In terms of understanding DNS basics, it’s probably note.

Name resolution in the DNS is walking a sequence of node names. The CNAME construct is for that node-walking process, but makes a jump, while doing that resolution, over to another name sequence. It’s a DNS construct, not an explicit DKIM construct. (This goes back to the importance of recognizing layers of functionality.)

With all of that, I’m ultimately not understanding “But that now creates some confusion, because CNAME is designed for name resolution and (unlike TXT eg) not database lookup.”

5

Dave,

Thanks for these clarifications. Let’s get the easy things out of the way:

  1. Yes, I should have said address instead of route; that was sloppy.
  2. Yes, a database has more functionality than lookup, but I was using that term to contrast one concept with another. More on that below.

You’re right in noting that DNS is best thought of as just mapping names to values; and those names are often hostnames but don’t need to be. My observation is that this was not the original intent of DNS. As Wikipedia explains:

The resource records contained in the DNS associate domain names with other forms of information. These are most commonly used to map human-friendly domain names to the numerical IP addresses computers need to locate services and devices using the underlying network protocols, but have been extended over time to perform many other functions as well.

My interest is in this extension, and how it has design features in common with many extensions we make to software. The particular kind of extension that I’m focusing on is what I call overloading by piggybacking, in which you have a unit of functionality (which I call a concept) that serves a particular purpose, and you extend it to support a new purpose which is mostly but not completely compatible with it. For exactly the kinds of reasons you note (in this case, that getting support for new DNS records has “an extremely problematic history”), it is often preferable to stick with the piggybacking and pay the costs rather than attempt a cleaner redesign.

My post attempts to lay out the details of the piggybacking in this case, and argued as follows:

  1. The initial concept of DNS (c. 1983) was to translate hostnames to addresses.
  2. At some later point, it was realized that it would be useful to use DNS for other purposes, piggybacking on its ability to store attributes for hostnames. This led to the TXT record (c. 1993) which allowed you to store arbitrary values against a hostname.
  3. The TXT record mechanism had some flaws; in particular, it was so useful that TXT records became big and unwieldy, so people looked to finding ways to get a more limited lookup. This (and correct me on this if I’m wrong) is where keys containing underscores were helpful: they let you get back a smaller set of TXT records by looking up not foo.com but _mykey.foo.com.
  4. As protocols like DKIM evolved, people realized they could use these keys even for the records in DNS that had been designed for address lookup, so you could create a CNAME record, for example, that mapped _domainkey.foo.com to the hostname of a machine that holds the public key for foo.com’s DKIM service.

My theory of concepts says that a tradeoff like this inevitable has some negative consequences. I hypothesized that the fact that some DNS providers would not allow underscores in CNAME keys was one such consequence. But I can’t imagine that there aren’t others. Do you disagree?

The alternative, which I agree seems infeasible given the difficulty of changing something as deeply embedded as DNS, would be not just a new record type, but a new kind of lookup, where you have two arguments: the host name (foo.com) and the property (domainkey). Does that make sense?

6

Daniel, I fear that ‘initial use’ is confused with ‘overall intent’. Yes, initial use was mapping a name to an address. I’d guess it is still by far the dominant use. But the statistics of use don’t determine the intent of the design. They merely make a case for basic efficacy.

Since its inception, the DNS has been the target of all sorts of additional uses. This gives credence to a view that the ‘intent’ was to permit other uses. (Not proof of it, but still…)

As for overloading by piggybacking, the description of the concept you provide is reasonable, but I’ll suggest it does not apply here. DNS was designed for a few different kinds of extensibility. It is not ‘overloading’ to use that. It’s just… extending.

An example of extending in a way that appears not to have been anticipated is, indeed, the underscore hack, which is a means of creating a clean mechanism for defining attributes to a name. Absent that, DNS node names have no type or other attribute information. While the underscore hack is ugly, it’s also effective. (cf, DKIM as an overlay for key distribution. Ugly but effective. And remarkably simple, compared with, say, DANE.)

“My post attempts to lay out the details of the piggybacking in this case, and argued as follows:”

You might want to review RFC 1034 carefully. I think it makes clear that the DNS was intended for a much broader range of uses than just name to address mapping.

7

Hi Dave,

I had a chance to read many of the DNS RFCs carefully, including a crucial one that you modestly didn’t mention but which you authored yourself, and which explains the need for a registry of underscore labels.

I’ve updated my post with some notes at the bottom summarizing what I learned, and refining my explanation of where the overloading is. Clearly you’re right that the “underscore hack” as you call it is effective. What interests me from a design perspective is what costs there might be to pay for this, and whether it’s a symptom (as my original post suggested) that DNS itself might have offered more flexible key/value lookup functionality.

Daniel

8

Daniel, hi. A few responses:

  1. Cost of the _underscore hack: Seems to reasonably modest. Implementation cost at the creator is being able to create an _underscored subdomain and adding a TXT record to it. More difficult than it should be, with the tools some DNS providers offer, but easy with others. The other implementation cost is the DNS query by a receiver. That seems to be only one extra lookup – which includes traversals down one or a few subdomain names. (You note the possibility of multiple, competing TXT records, which I’ll comment on, below) All in all, the operational DNS world does not seem to view any of this as an interesting increase in overhead. 20 or 30 years ago, I think it did.

  2. For reference, the publication of RFC 8552 (on the use of underscored names) came more than 10 years after the first draft was written. There were some basic technical problems with the early versions of the draft but more importantly there was very strong resistance from the core DNS technical community and it killed work at that point. Publication was eventually possible only because of changes in that community (including one of its number pressing for the work to re-start.)

  3. Confusion. You cite CSNet phone numbers in the DNS. I developed and ran the CSNet email service for its first couple of years and the DNS didn’t exist yet; we used an independent (rather small) file with the phone numbers. That said, I didn’t track later CSNet developments. While I can imagine some ways to use the DNS for the phone numbers, note that the places being referred to, by a phone number, were not on the IP Internet, where the DNS was/is. They were only accessible by an email gateway. I don’t recall hearing about CSNet phone numbers in the DNS; do you have a reference to this? (I do know there was a major effort to get several independent services, including CSNet, to standardize on using the DNS, but I think that wasn’t specifically tied to phone numbers.)

  4. Small quibble. In your update, you refer to the DNS as a database. In casual discussion, that’s a fine, generic, non-technical term. In technical discussions, it can create confusion. Typical ‘database’ systems have ‘searching’ and one or another kind of fuzzy string matching. The DNS doesn’t do this. You must give it only and exactly a complete lookup string to use (in classic key/value terms.) Hence, it is safer to refer to the DNS as a lookup service.

  5. Competing TXT records. You cite the challenge of parceling out the type of content being sought, among a collection of TXT records that contain different types of information. This is the long-standing concern about TXT records, since they do not overtly distinguish the ‘type’ of information they contain, the way a typical DNS RR will do. However this is exactly the problem that the _underscore hack mostly solves: The TXT record(s) put under and _underscored name are specific to the ‘scope’ of that _underscored name. I say “mostly” because, of course DNS wildcard names can propagate TXT records down its the subtree, including into _underscored nodes.

  6. CNAME. Hmmm. You seem to indicate that CNAME use aids in the competing-TXT records problem. That does not match my sense of its use. Rather, I understand CNAMEs to be use for administrative convenience, rather than in aiding the work of the side doing lookups.

  7. ‘Intentional’ names. This is a challenging topic, because it requires parceling out – and juggling – a number of issues. A core point is that domain names, per se, are typeless and semantics free. A name is only and exactly a series of strings that have no semantics to the core DNS functions, other than for traversing the tree. Traversal does not depend on type or semantics. This was true in the original DNS and remains true. Really!

Semantics and typing are higher-level constructs. That is, they are layers built on top of this core, but do not change that core. The eventual use of www.* or imap.* and the like, for distinguishing access to a service, is an operational convention. It actually has nothing to do with the DNS, itself, but with the way some/many sites choose to administer their own use of the DNS.

I think I understand your phrasing of “the ability to perform a lookup on two keys”, given the example you use – _ldap._tcp.foo.com – but really, these are not ‘keys’ in the sense the term is used for database queries. And certainly the DNS itself knows nothing about the conventions being applied in the use of those. And I continue to be uncomfortable with calling any of this ‘overloading’ since it implies competing or confusing system semantics, and this has none of that.

1 Like
9

Hi Dave @dcrocker,

Thanks so much for your thoughtful response, and sorry for the slow reply. I had a busy end of term and then a trip to Germany and so I’m only now finally catching up on this.

Everything you say seems reasonable. A few small things:
– The reference to CSnet phone numbers was in RFC882
– I see your point about the word “database” but I’m not sure I want to reserve it for systems that have elaborate query languages, since we don’t have a better term that I can think for something that provides key/value lookups (eg). Note also that RFC882 uses that term: “What is needed is a distributed database.”
– You’re absolutely right about CNAME and my misunderstanding of that; I clarified in my note that it’s about introducing an extra level of indirection for delegation (eg to allow easier key rotation).

About my use of the term “overloading”:
– In concept design, overloading means using one concept for multiple purposes (often purposes that are associated with other distinct concepts); it doesn’t have the programming-language sense of the semantics of a term being uncertain until it’s resolved in some context.

About the general approach and what I’m trying to do here:
– The idea of concept design is to clarify what’s going on in a design, and what tradeoffs are involved. I think perhaps you’re reacting to the criticism implicit in my original post that underscore labels were an unprincipled hack. I didn’t mean to give that impression! On the contrary, I think my analysis shows that there is an interesting tradeoff here. As I’ve learned more from you and from deeper reading, I’ve come to appreciate that tradeoff better, and now understand that it’s in large part about avoiding changes to DNS and its resource types. Whether that’s the right way to go (and you’re a better judge of that than me), the analysis at the very least points out the nature of this tradeoff and some of the costs.

I’ve revised (again!) my original post with a more careful analysis of the concepts at play. For completeness in this forum, I’m appending the update below.

Best,

Daniel

Update

A few updates and corrections, following some further investigation and input from DNS experts:

DNS as a general database. The earliest RFCs mention DNS holding information beyond host addresses—including phone numbers for CSNET, for example—and make it clear that the resource records were not to be limited to the initial types. It wasn’t until later, though, that the idea of DNS as a general key/value store seems to have emerged explicitly. Jerry Saltzer, who developed a name service for Athena at MIT called Hesiod, told me that Paul Mockapetris added the TXT resource type to support more general lookups, as required by applications such as Hesiod.

Domain names as intentional names. Domain names that included property labels go back at least to Hesiod, which used an @-symbol to separate the property-specifying part from the rest, eg. finger-server@berkeley.mit.edu. A project at MIT in 1999 explored this general idea, in which a name does not designate a service directly, but rather specifies the properties for a desired service, and called it intentional naming. In 2000, RFC 2782 described the addition of the SRV resource type, which mapped domain names of the form _service._protocol.name to server/port names, allowing intentional names such as _ldap._tcp.foo.com.

Domain names that include property keys. A domain name like domainkey.foo.com is not an intentional name that specifies a service. The DKIM protocol does not require a service; all that’s needed is for the DKIM key to provided for the domain. Instead, this domain name is a combination of a domain name (foo.com) and a key to be looked up in the DNS records of that domain name.

Underscores in domain names. The use of underscores in these extended forms of name prevented conflicts with hostnames, but introduced the new risk of the new labels conflicting. In 2019, RFC 8552 described the convention of naming with underscored labels, and introduced a registry to avoid collisions.

Underscore confusions. The early RFCs said that domain names should be as general as possible, but confusing wording misled many people. A much-quoted statement from RFC 882 seems to say that underscores are not permitted in the labels that comprise domain names: “The labels must follow the rules for ARPANET host names. They must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen.” This statement, however, seems on closer reading to be an informal explanation for a grammar that is not intended to be mandatory: “The preferred syntax of domain names is given by the following BNF rules. Adherence to this syntax will result in fewer problems with many applications that use domain names (e.g., mail, TELNET).” This complicated position is elaborated in RFC 1035 which states:

The DNS specifications attempt to be as general as possible in the rules
for constructing domain names. The idea is that the name of any
existing object can be expressed as a domain name with minimal changes. However, when assigning a domain name for an object, the prudent user
will select a name which satisfies both the rules of the domain system
and any existing rules for the object, whether these rules are published
or implied by existing programs.*

Not surprisingly this has confused even experts; a ballot amongst a consortium of companies voted to sunset the use of underscores in DNS names appearing in certificates, citing the statement in 1035 that “labels must follow the rules for ARPANET host names” which it took, incorrectly, to specify “the characters which may be used in DNS domain names.”"

CNAME records for DKIM. Use of CNAME resource records for DKIM is not (I believe), as I originally suggested above, to avoid the use of TXT records, but rather to provide an extra level of indirection so that a domain can delegate to a hosting service the job of assigning (and rotating) DKIM keys. The use of the _domainkey prefix alone would limit the number of TXT records returned, since the extended domain name has a different set of resource records associated with it than the base domain name.

Refining the concept design analysis. In summary, there are (at least) three distinct concepts in play here. First there is the concept of a HierarchicalName, which allows a name space to be divided into separately managed zones. This concept is familiar from file systems, and from the structure of many web APIs (which use so called “RESTful” names for resources). Second is the concept of IntentionalName, in which a name becomes a kind of specification. This also existed prior to DNS, although the concept has been less widely adopted. Third is the concept of Metadata in which an object has a collection of properties associated with it; a photo, for example, has its capture time and exposure; a file has its creation time; a DNS domain has its domain key.

The lens of concept design helps us recognize that much of the richness of DNS that we have explored comes from the fact that three distinct concepts are being offered. The familiarity of these existing concepts should make DNS easier to understand.

What is unusual is that all three concepts are implemented by the same mechanism. In concept lingo, the second and third concepts are “piggybacked” onto the first concept, with the properties and specifiers of Metadata and IntentionalName respectively both represented as labels in a prefix of a domain name. Like many piggybacking designs, this is ingenious and solves some problems. In particular, it allowed DNS itself to remain unchanged, without the need for new resource types or mechanisms (although it did require the creation of a new registry to avoid name clashes).

The downside is that piggybacked concepts generally cannot be fully supported by a mechanism that was not designed for them. A full implementation of IntentionalName, for example, would allow wildcard specifiers, so that for example one could request not only a color printer _printer._color.local.foo.com or a monochrome printer _printer._mono.local.foo.com but also any printer _printer.*.local.foo.com whether color or monochrome. As noted in RFC8552, DNS cannot support such wildcards. Another price paid for the piggybacking is some additional complexity involved in squeezing the new functionality into a Procrustean bed—here, the underscore, and all the confusion created about whether it is permitted.

Whether the DNS design is bad or good is not the main issue here—and now I have a better appreciation of the tradeoffs I am less inclined to insist that this piggybacking is a mistake. What my analysis shows, I hope, is that concept design can reveal the underlying issues and make clearer whatever tradeoffs are being made.

[end]