Service Discovery in Production: DNS, Registry, or Mesh — Which One Won’t Let You Down?

Every microservices tutorial skips the part where your ten services need to find each other at runtime, when IPs are ephemeral, containers restart every few minutes, and your load balancer config is already three commits out of date. That’s the service discovery problem, and it’s where architectures go to die.

There are three dominant patterns: DNS-based, registry-based, and service mesh. Each has a real use case. Each will wreck you in specific scenarios. This article walks through all three with working configs, so you can make an actual decision instead of just picking whatever the blog post you found last week recommended.

---

Why Static Config Is Already Dead

Back when you had five servers and an Nginx reverse proxy, you hardcoded the upstreams. proxy_pass http://192.168.1.10:8080. Done. It worked.

Now you have a fleet of containers. They live on dynamic IPs assigned by the orchestrator. They scale up and down. They get killed by health checks and replaced. The concept of a stable, pre-known address died with VMs. You need something that answers the question "where is service X right now?" at request time — not at deploy time.

That’s service discovery. The three approaches below differ in who holds that answer, how fresh that answer is, and what you have to build to use it.

---

Pattern 1: DNS-Based Discovery

The oldest trick in the book, and still genuinely good in the right context.

The idea: every service registers under a predictable DNS name. Clients resolve that name when they need to make a request. The DNS server (CoreDNS, Route53, or whatever you’re using) handles the mapping from name to IP(s).

In Kubernetes, this is the default. Every Service object gets a DNS entry like my-service.my-namespace.svc.cluster.local. CoreDNS runs as a cluster addon and answers those queries. You don’t configure it — it just works.

Simple CoreDNS config (for a custom zone outside K8s):

# Corefile
.:53 {
    # Forward queries for internal zone to etcd backend
    etcd internal.myapp.local {
        path /skydns
        endpoint https://cd-linux.club:2379
    }

    # Forward everything else upstream
    forward . 8.8.8.8 8.8.4.4

    cache 30
    log
    errors
    health
}

Registering a service into etcd for CoreDNS SkyDNS format:

# Register api-service at 10.0.1.55:8080
etcdctl put /skydns/local/myapp/internal/api-service/node1 \
  '{"host":"10.0.1.55","port":8080,"ttl":30}'

Your client code just does http://api-service.internal.myapp.local:8080 and DNS handles the rest. Multiple A records = basic round-robin load balancing for free.

Gotchas

TTL caching is your enemy. Most DNS clients cache records for the TTL duration. If a pod dies and its replacement gets a new IP, clients might keep hitting the dead address for 30 seconds. In Kubernetes this is mostly handled by the control plane invalidating records, but in home-rolled setups you’ll feel this pain directly.

No health checking. Pure DNS has no concept of "is this backend actually healthy?" You can have a perfect DNS record pointing to a crashed service. Clients will fail until the record expires or you delete it manually.

Round-robin isn’t load balancing. DNS round-robin distributes connections, not load. A slow backend will accumulate connections. You get none of the sophistication — no weighted routing, no circuit breaking, no retries.

DNS doesn’t handle ports well. SRV records exist for this, but almost nothing uses them correctly. Most clients just do A record lookup and rely on a convention for the port. That’s fragile.

When DNS-Based Discovery Actually Makes Sense

• Kubernetes internal traffic where you’re happy with kube-proxy
• Simple microservice setups with stable services and acceptable TTL lag
• Cross-datacenter routing where you control the DNS zones
• When you genuinely don’t want to run another stateful component

---

Pattern 2: Registry-Based Discovery

This is the step up when DNS isn’t enough. A registry is a dedicated service that maintains a real-time catalog of what’s running and where. Services register themselves on startup, send heartbeats, and deregister on shutdown. Clients query the registry to get a fresh list of healthy backends — then connect directly.

Consul is the gold standard here. etcd works but lacks the health check primitives. Eureka (Netflix, Java ecosystem) is still around but showing its age.

Consul in Practice

The architecture: Consul agents run on every node (or container). Services register with the local agent. Agents gossip state across the cluster using the Serf protocol. Clients query the Consul HTTP API or DNS interface for healthy instances.

docker-compose.yml for a Consul cluster (3-node):

version: "3.8"

services:
  consul-server-1:
    image: hashicorp/consul:1.17
    command: >
      agent -server
      -bootstrap-expect=3
      -node=consul-server-1
      -bind=0.0.0.0
      -advertise=172.20.0.2
      -client=0.0.0.0
      -ui
      -retry-join=172.20.0.3
      -retry-join=172.20.0.4
    networks:
      consul-net:
        ipv4_address: 172.20.0.2
    ports:
      - "8500:8500"  # HTTP API + UI
      - "8600:8600/udp"  # DNS

  consul-server-2:
    image: hashicorp/consul:1.17
    command: >
      agent -server
      -bootstrap-expect=3
      -node=consul-server-2
      -bind=0.0.0.0
      -advertise=172.20.0.3
      -client=0.0.0.0
      -retry-join=172.20.0.2
    networks:
      consul-net:
        ipv4_address: 172.20.0.3

  consul-server-3:
    image: hashicorp/consul:1.17
    command: >
      agent -server
      -bootstrap-expect=3
      -node=consul-server-3
      -bind=0.0.0.0
      -advertise=172.20.0.4
      -client=0.0.0.0
      -retry-join=172.20.0.2
    networks:
      consul-net:
        ipv4_address: 172.20.0.4

networks:
  consul-net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.20.0.0/24

Service registration via Consul agent API:

# Register a service with a health check
curl -X PUT https://cd-linux.club:8500/v1/agent/service/register \
  -H "Content-Type: application/json" \
  -d '{
    "ID": "api-service-1",
    "Name": "api-service",
    "Tags": ["v2", "primary"],
    "Address": "10.0.1.55",
    "Port": 8080,
    "Check": {
      "HTTP": "http://10.0.1.55:8080/health",
      "Interval": "10s",
      "Timeout": "3s",
      "DeregisterCriticalServiceAfter": "30s"
    }
  }'

Querying for healthy instances:

# Get healthy instances of api-service
curl "https://cd-linux.club:8500/v1/health/service/api-service?passing=true" | \
  jq '.[].Service | {address: .Address, port: .Port}'

Your application does this query at startup and periodically, caches the result, and implements its own client-side load balancing. Libraries like github.com/hashicorp/consul/api (Go) or python-consul2 handle the polling for you.

Sample Go snippet for Consul-backed client:

package discovery

import (
    "fmt"
    "math/rand"

    "github.com/hashicorp/consul/api"
)

type ConsulDiscovery struct {
    client      *api.Client
    serviceName string
}

func NewConsulDiscovery(addr, serviceName string) (*ConsulDiscovery, error) {
    cfg := api.DefaultConfig()
    cfg.Address = addr
    client, err := api.NewClient(cfg)
    if err != nil {
        return nil, err
    }
    return &ConsulDiscovery{client: client, serviceName: serviceName}, nil
}

// PickEndpoint returns a random healthy backend address
func (d *ConsulDiscovery) PickEndpoint() (string, error) {
    entries, _, err := d.client.Health().Service(d.serviceName, "", true, nil)
    if err != nil {
        return "", err
    }
    if len(entries) == 0 {
        return "", fmt.Errorf("no healthy instances of %s", d.serviceName)
    }
    e := entries[rand.Intn(len(entries))]
    return fmt.Sprintf("%s:%d", e.Service.Address, e.Service.Port), nil
}

Gotchas

Split-brain during Raft elections. Consul uses Raft for leader election. If you lose quorum (e.g., 2 of 3 servers go down), the cluster enters a read-only state. Services can’t deregister. New registrations fail. Always run an odd number of servers (3 or 5) and keep them in separate failure domains.

Deregistration lag. If your service crashes hard without calling the deregister API, Consul waits for DeregisterCriticalServiceAfter before removing it. During that window, some clients will hit a dead backend. Set this aggressively (30s) and implement proper retries in your clients.

Registry is now a critical dependency. You’ve introduced a stateful cluster that must stay up for your services to find each other. Budget accordingly for monitoring, backups, and failure drills.

Client-side load balancing complexity. You’re now writing or adopting LB logic in every service client. Getting weighted routing, circuit breaking, and retry logic right across five different language ecosystems is genuinely hard.

When Registry-Based Discovery Makes Sense

• Multi-datacenter setups where you need real-time health data
• When you need service metadata and tagging (for canary routing, versioning)
• Non-Kubernetes environments: bare metal, VMs, mixed stacks
• When you want the registry to also serve as your KV store and config backend

---

Pattern 3: Service Mesh

If DNS is a hammer and a registry is a drill, a service mesh is a whole power tools workshop that you also have to assemble yourself, calibrate, and maintain. It’s powerful and it has a real price.

A service mesh injects a sidecar proxy (typically Envoy) into every service pod. All traffic between services goes through these proxies. The proxies handle discovery, load balancing, retries, circuit breaking, mTLS, tracing, and rate limiting — without touching application code. A control plane (Istio’s Istiod, Linkerd’s controller) manages the proxy configs centrally.

Discovery works because the control plane has full knowledge of the cluster state (via the Kubernetes API) and programs each Envoy sidecar with the current endpoint list. No DNS TTL lag. No registry to query. The proxy knows.

Istio Setup Sketch (K8s)

# Install with istioctl
istioctl install --set profile=default -y

# Enable sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled

# Verify mesh status
istioctl analyze -n production

VirtualService for traffic splitting (canary deployment):

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-service
  namespace: production
spec:
  hosts:
    - api-service
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: api-service
            subset: v2
    - route:
        # 90% stable, 10% canary by default
        - destination:
            host: api-service
            subset: v1
          weight: 90
        - destination:
            host: api-service
            subset: v2
          weight: 10
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: api-service
  namespace: production
spec:
  host: api-service
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 1000
        maxRequestsPerConnection: 10
    outlierDetection:
      # Circuit breaker: eject hosts with 5 consecutive 5xx
      consecutiveGatewayErrors: 5
      interval: 10s
      baseEjectionTime: 30s
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

Linkerd is the simpler alternative. If you don’t need Istio’s full feature set, Linkerd v2 is dramatically easier to operate. It uses a Rust-based micro-proxy instead of Envoy, has a tiny control plane, and the default install just works.

# Install Linkerd
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd check

# Inject into a deployment
kubectl get deploy -n production -o yaml | \
  linkerd inject - | \
  kubectl apply -f -

Gotchas

The resource cost is real. Every sidecar proxy consumes CPU and memory. At small scale, this is measurable. A Envoy sidecar at idle takes ~50MB RAM and has nonzero CPU overhead on every request. Multiply by 200 pods.

Debugging gets harder, not easier. When something breaks, you’re now debugging application code plus sidecar config plus control plane state. istioctl proxy-config and Kiali help, but the learning curve is steep.

Upgrade pain. Istio has a history of breaking changes between minor versions. Upgrading a mesh that’s carrying production traffic requires a careful canary rollout of the control plane itself. This is not a weekend project.

mTLS bootstrapping complexity. The mesh manages certificates automatically, but if you mix mesh and non-mesh services (legacy VMs, external databases), you need to handle PeerAuthentication policies carefully or you’ll find services silently rejecting connections.

Latency. An extra proxy hop per request adds latency. Usually sub-millisecond, but it’s real and it compounds across deep call chains. Benchmark your p99 before and after mesh adoption.

When Service Mesh Makes Sense

• Kubernetes-native architecture with 10+ services
• You need mTLS between services without touching application code
• Advanced traffic management: canary, A/B, fault injection, rate limiting
• You have a dedicated platform team to own it
• Compliance requirements that mandate encryption in transit everywhere

---

Side-by-Side Comparison

Factor	DNS	Registry (Consul)	Service Mesh
Freshness	TTL lag	Near real-time	Real-time (control plane push)
Health checking	No	Yes (native)	Yes (proxy-level)
LB sophistication	Round-robin only	Client-side (you build it)	Full (weighted, circuit break)
Ops complexity	Low	Medium	High
Language agnostic	Yes	Mostly (via DNS interface)	Yes (sidecar)
Non-K8s support	Yes	Yes	Painful
mTLS	No	Consul Connect	Yes (native)
Resource overhead	Minimal	Consul cluster	Sidecar per pod

---

Production-Ready Best Practices

Layer, don’t replace. DNS is still your fallback layer. Even with Consul or a mesh, DNS is how humans and monitoring systems reach things. Keep it coherent.

Health check depth matters. A shallow HTTP 200 from /health isn’t enough. Your health endpoint should check database connectivity, downstream dependencies, and critical cache availability. A service that’s "up" but can’t reach its database is not healthy.

Implement retries at every layer. Service discovery tells you where to go. It doesn’t guarantee the request will succeed. Build retry logic with exponential backoff and jitter into every service client, regardless of which discovery pattern you use.

Monitor the discovery layer itself. Consul leader elections, CoreDNS query error rates, Istio pilot push latency — these are your canary metrics. Set up alerts before you need them. A broken discovery layer looks like a random cascade of partial failures across your entire system.

Test failure modes deliberately. Kill your Consul leader. Block sidecar-to-control-plane communication. Run a DNS TTL expiry simulation. You need to know how your system degrades before production teaches you.

---

The Honest Decision Framework

If you’re running Kubernetes and have fewer than 15 services: use Kubernetes Services + CoreDNS. It’s already there, it works, and you don’t need more complexity.

If you’re running mixed infrastructure (VMs + containers, multi-datacenter) or need real-time health information: Consul. It handles the heterogeneous case better than anything else and the operational cost is manageable.

If you’re on K8s, have a platform team, and need traffic management beyond what kube-proxy offers — canaries, circuit breaking, mTLS everywhere: service mesh. Prefer Linkerd if you want something maintainable. Use Istio if you specifically need its feature set and are prepared to own it.

The mistake most teams make is jumping straight to a service mesh because it looks impressive on an architecture diagram. They inherit the complexity without having exhausted simpler options first. Run the boring solution until it demonstrably breaks, then upgrade. Discovery infrastructure is not the place to show off.