An Overview of Distributed Service System Design
Overview and definition
A distributed system is a system made up of a group of independent computers (nodes) working together over a network. Together they accomplish a business goal and present themselves to users as a "single system". From the user's point of view, a distributed system should be as transparent as possible: users shouldn't need to know which machine handled a request or where the data lives, and they shouldn't notice nodes being added, removed, or migrated.
In one sentence:
Distributed system = multiple independent computers, cooperating over a network, that look like one system from the outside
Two points to remember:
- Multiple nodes cooperate to achieve the same business goal.
- From the outside, it behaves as much like a single whole as possible (transparency).
Note: distributed system design often requires trade-offs among consistency, availability, and partition tolerance (CAP). In practice, "eventual consistency" is widely adopted in exchange for high availability and scalability.
Compared with a centralized C/S architecture
A centralized (monolithic) C/S architecture:

A typical distributed system:

Why distributed systems are "hard"
Every assumption from single-machine systems breaks down in a distributed setting 👇
On a single machine you assume:
- Memory is reliable
- Clocks are consistent
- Calls always succeed
In a distributed system, reality is:
- The network can fail
- Nodes can crash
- Clocks are out of sync
- Messages can be lost / duplicated / delayed
👉 Distributed system = building a reliable system in an "untrustworthy environment"
The core problems a distributed system must solve
1️⃣ Communication
How do nodes talk to each other?
- HTTP / RPC / gRPC
- Message queues (Kafka / RocketMQ / RabbitMQ)
2️⃣ State consistency (the most central)
How do you keep data consistent across multiple machines?
- Strong consistency (bank transfers)
- Eventual consistency (likes, recommendations)
👉 This leads directly to the CAP theorem
3️⃣ Fault tolerance
What happens when a machine goes down?
- Replicas
- Automatic failover
- Retries / timeouts
4️⃣ Scalability
What if traffic grows 10×?
- Add machines (horizontal scaling)
- Load balancing (load balancer)
The CAP theorem (the "law of physics" of distributed systems)
When a network partition (P) occurs, you can only choose one of C and A
- C (Consistency)
- A (Availability)
- P (Partition tolerance): mandatory
Real-world systems:
- Financial systems → CP
- Social / recommendation → AP
Design overview
- Message-driven: treat messages as the "first-class citizen" of inter-service collaboration; naturally decoupled and easy to scale.
- Actor model: model service instances as Actors that process messages and maintain their own state, isolating concurrency.
- MongoDB/Postgres: models that fit business objects, with support for sharding and partitioning, which helps scalability and high availability.
- Redis: a high-speed cache for read/write hotspots; can also handle rate limiting, short-lived queues, and similar roles.
- Kafka/RocketMQ: message channels between producers and consumers.
- Engineering principles: clear boundaries, single responsibility, idempotency and retries, and statelessness wherever possible (especially at the gateway layer).
The Actor model
In practical engineering terms, the theoretical Actor corresponds to a microservice submodule with its own state.
See Wikipedia: Actor Model
An Actor is a basic unit of computation:
- It receives messages (Mailbox).
- It processes messages based on its current state and behavior (Behavior + State).
- It produces a new state and, optionally, response messages.
Actors are isolated from each other, share no memory, and cooperate purely through messages. This model is especially well suited to highly concurrent, strongly isolated distributed scenarios.

In the system's layering, "Service N" can be modeled as a group of Actors: each service is split into several single-responsibility Actors handling sub-domain logic such as sessions, orders, inventory, and settlement. This reduces coupling and makes scaling easier.
State machines (FSM) and Actors
Logic inside many Actors can be organized as a finite state machine (FSM). The four elements of an FSM:
- State
- Event
- Action
- Transition
A state machine starts from the "initial state", moves to the next state when an event fires and performs the corresponding action, and eventually reaches the "final state". Treating the state machine as a "black box" that exposes only events and results achieves good isolation of internal information and state safety.
Data layer: MongoDB/Postgres
MongoDB is a very good fit for distributed systems:
- Its data model (Document/Collection/Table) is close to the object structure of the business, making reads and writes natural.
- Sharding is built for horizontal scaling, distributing data evenly across nodes by key.
- Replica sets provide high availability and read/write splitting strategies.
- Consistency and transactions: multi-document transactions are supported, but in practice "eventual consistency + idempotent compensation" is often used to reduce coupling and overhead.

Practical recommendations:
- Design the shard key carefully (high cardinality, even writes, high query hit rate).
- Base read/write strategy on replica set roles (write to primary, read from secondaries).
- Use sensible indexes and TTL indexes to manage hot and historical data.
Use an ORM well; in Go, gorm is very popular.
Cache layer: Redis
Redis's role in this architecture:
- Hot data cache: reduces database load and improves read performance (using strategies such as Cache-Aside / Read-Through / Write-Through).
- Short-lived queues and rate limiting: simple queues and rate limiting based on List / Stream / token bucket.
- Sessions and transient state: store short-lived sessions, idempotency tokens, distributed locks, etc.

Typical risks and countermeasures:
- Cache penetration: cache null values for non-existent keys + a Bloom filter.
- Cache breakdown (hot-key expiry): use mutex-guarded refresh for hot keys, or logical expiry + background refresh.
- Cache avalanche: randomize expiration times, and set up multi-level rate limiting and degradation strategies.
Service discovery and configuration: etcd
etcd is a distributed key-value store based on the Raft consensus algorithm, commonly used for service registration and discovery, dynamic configuration, and distributed coordination:
- Service registration and discovery:
- Registration: instances write their address and metadata under
/services/{name}/{instanceId}with a lease (Lease + TTL) and send periodic KeepAlives. - Discovery: the gateway or callers Watch the corresponding prefix to detect instances coming online or going offline, and update local routing.
- Robustness: keys are deleted automatically when the lease expires, avoiding "zombie instances"; clients should reconnect with exponential backoff and jitter.
- Registration: instances write their address and metadata under
- Dynamic configuration:
- Organize configuration under the
/config/{app}/{env}/prefix; publishers write, clients Watch and hot-reload. - Split large configurations into modules / load them in segments; sensitive configuration needs encryption or a dedicated configuration center.
- Organize configuration under the
- Coordination:
- Distributed locks / leader election: use the etcd concurrency API (session-based Mutex/Campaign).
- Typical uses: sharded task scheduling, single-leader writes, centralized arbitration of rate-limit quotas.
- Operations:
- A cluster of 3 or 5 nodes is ideal (odd numbers for voting; too many nodes actually reduces write performance).
- Configure snapshots and compaction properly to avoid an oversized Watch backlog; set timeouts and heartbeat thresholds for high-latency links.
In this architecture, etcd plays the infrastructure role of "service registration and discovery + dynamic configuration + coordination primitives", supporting gateway routing and internal service addressing.
Communication and consistency strategy
- Communication: - Synchronous RPC (HTTP/gRPC) for small closed loops that need strong real-time behavior and strong consistency. - An asynchronous message bus for decoupling and smoothing traffic peaks (RocketMQ/Kafka/RabbitMQ/Redis Stream, etc.).
- Retries and idempotency:
- Retry when a downstream is briefly unavailable; every retryable write must be idempotent (dedup keys, idempotency tokens).
- Eventual consistency:
- Saga / Outbox pattern: use a local transaction + outbound events to guarantee changes are delivered, and orchestrate compensation on failure.
- Global identifiers:
- Snowflake / segment-cached ID generation produces sortable distributed IDs, useful for routing and auditing.
Message middleware: RocketMQ/Kafka
RocketMQ/Kafka are high-performance, low-latency distributed message middleware with complete messaging semantics and enterprise features:
- Core concepts: Topic, Tag, Message Key, Producer Group, Consumer Group.
- Delivery semantics: at-least-once by default; exactly-once has to be implemented at the business layer with idempotency keys / dedup tables.
- Ordered messages: local ordering within a queue based on a sharding key, suitable for order state flows, fund ledgers, etc.
- Transactional messages:
- Flow: send a half message (Prepare) → execute the local transaction → Commit/Rollback;
- The broker can check back on the transaction status (Transaction Check) to avoid dangling transactions; the producer must implement the check-back interface;
- Commonly used for reliable event publishing in Saga/Outbox.
- Delayed / scheduled messages: built-in multi-level delays; combined with retry policies to implement flexible transactions and peak shaving.
- Retries and dead letters:
- Failed consumption is redelivered per the retry policy; beyond the threshold, messages go to the DLQ (dead-letter queue) for troubleshooting and manual compensation.
- Consumption modes: clustering (load-balanced) and broadcasting (every consumer receives all); business consumers usually use clustering mode.
- Design recommendations:
- Idempotent messages: deduplicate with a business-unique key (such as an order number); consumers "check for duplicates first, then execute".
- Load and scaling: plan the number of queues based on throughput and ordering needs; avoid overly large messages (< 1MB recommended); put large objects in storage and only references in messages.
- Observability: enable message tracing and metrics reporting; watch backlog, latency, and check-back ratio.
In this architecture, RocketMQ/Kafka serves as the "event bus", supporting cross-service decoupling, peak shaving, and eventual consistency:
- Outbox: business table + outbox table committed in one local transaction, published to RocketMQ in the background, and stored idempotently on the consumer side.
- Saga: an orchestrator advances/rolls back steps based on events, with compensation messages on failure.
Topology and deployment
Typical topology:
- Gateway: the single external entry point, handling authentication, routing, rate limiting, and aggregation.
- Service registration and discovery: an etcd-based registry with Watch, providing dynamic addressing and health checks (optionally with sidecars).
- Internal services: domain-partitioned microservices / Actor groups that interact via RPC/messages.
- Data and cache layer: a MongoDB cluster + Redis cluster, designed with replica sets, Sentinel, and sharding.
- Messaging layer: a RocketMQ broker cluster + NameServer, scaling throughput by Topic/queue.

When it fits
- Large numbers of concurrent connections (massive user bases, IoT devices, game gateways, etc.).
- Large data volumes requiring horizontal scaling and elasticity (logging / order / content platforms, etc.).
- Hard high-availability requirements (parts of financial payments, read paths of core transactions, etc.).
Summary
This "message-driven + Actor + MongoDB/Redis" combination has a clear engineering path and a mature ecosystem: when performance falls short, scale out by adding nodes; when data grows, scale with MongoDB sharding; when read/write load is uneven, use Redis caching and rate limiting to shave peaks. The real difficulty is not "whether it works", but "whether it can be scaled and operated gracefully".
Next, in a follow-up post, I'll combine concrete design tools and code to build an end-to-end example of a "micro distributed service system", covering the minimal loop of service decomposition, communication, caching, consistency, and observability.