CAP Theorem
CAP theorem states that a distributed system can provide at most two of three guarantees: Consistency (all nodes see the same data at the same time), Availability (every request gets a response), and Partition Tolerance (system continues operating when network partitions occur).
CAP = planning a party with 3 friends: Consistent info, Available hosts, Partition-tolerant (works if some can't talk). The network WILL glitch (P is mandatory), so pick: does everyone get the same answer (C), or does everyone answer the phone (A)?
Network partitions are unavoidable in distributed systems — you can't choose to not be partition tolerant. So the real choice is: during a partition, do you prioritize consistency (CP: reject writes or return errors until partition heals) or availability (AP: accept writes and serve reads, accepting that nodes may have different data)? CP systems: ZooKeeper, etcd, HBase, Spanner. AP systems: Cassandra, CouchDB, Riak, DynamoDB (with eventual consistency).
CAP is widely misapplied. 'Consistency' in CAP is linearizability (reads see the most recent write), which is stronger than ACID's 'C' (data integrity constraints). 'Availability' in CAP means every non-failing node responds, which is stronger than practical availability (SLA uptime). PACELC extends CAP: during normal operation (no partition), the trade-off is between Latency and Consistency. Spanner accepts higher latency to achieve consistency via TrueTime. Cassandra accepts lower consistency (read repair, anti-entropy) for lower latency. In practice, systems tune consistency per-operation: DynamoDB's ConsistentRead flag, Cassandra's per-query consistency level (ONE, QUORUM, ALL). A QUORUM write (majority of replicas must acknowledge) + QUORUM read guarantees reading the latest write without requiring ALL replicas to be up.
I'd present CAP not as a static global choice but as a per-operation tuning knob. For the payment confirmation path, I'd use strong consistency (QUORUM or ALL reads) because showing a stale balance is a business problem. For the activity feed, I'd use eventual consistency (ONE read) because 100ms of staleness is acceptable and latency matters more. I'd design the system so that the consistency level of each operation is a conscious decision documented in the API contract.
Treating CAP as a permanent architectural choice you make once. Modern distributed databases (Cassandra, DynamoDB) let you tune consistency per request. Design your system to use the weakest consistency level that meets the correctness requirement for each operation.