The first step in any combat is to determine which creatures are surprised, if any. Surprise is a significant tactical advantage, so smart adventurers do everything then can to avoid being surprised and to surprise their foes when possible.
For the full text of the rules, see page 190 of the Player’s Handbook.
Determining which creatures are surprised can be difficult, but a brief examination of the rules text can simplify things. The rules make up just three paragraphs; a paragraph of explanatory text, a paragraph about determining surprise, and a paragraph about what happens when you’re surprised.
The second paragraph can be a bit confusing, but the first and last sentences are the important part.
The DM determines who might be surprised.
The DM is the final arbiter of the rules at the table, and this line reiterates that point. There may be some debate and disagreement, but the DM gets the final say. You might suggest that your character is surprised if you feel that you were surprised, and as a DM I would consider that a polite and respectable thing to do, but the DM still gets final say regardless.
Any character or monster that doesn’t notice a threat is surprised at the start of the encounter.
This sentence is a bit dense, so we’ll go through it piece by piece.
“Any character or monster” is the same as saying “any creature”. The Player’s Handbook tries to draw a distinction between characters and monsters, even though there is no actual mechanical difference.
“doesn’t notice a threat” has some room for interpretation, and I think it’s the most important part of the whole paragraph. A creature must “notice a threat”, meaning first that they only need to notice one threat, and second that they must notice that the threat is indeed a threat. That’s clear as mud, so let’s look at an example:
Four adventurers are walking along a forest road, enjoying the weather. The party’s ranger notices a wolf peering at them from the roadside. The ranger, comfortable with the wild beats of the world, gestures politely to the wolf and continues on his way. The party’s cleric notices a goblin on the opposite side of the road with a blade in its hand and a sadistic grin. The cleric raises his voice to alert his friends, but it’s too late: a group of goblins and their trained wolf pounce from their hiding places, and combat begins.
In this case, the cleric is probably the only member of the party who is not surprised because the cleric noticed a goblin and correctly identified it as a threat. The ranger noticed the wolf, but wrongly assumed that the wolf was simply going about its wolf business (huffing and puffing and didn’t pose a threat to the party.
The ranger might argue that they would have noticed something odd or threatening about the wolf’s behavior, and the DM is free to decide as they see fit. The DM might decide one way or the other, or they might allow the ranger to make a Wisdom (Insight) or possibly a Wisdom (Animal Handling) check, and use the result to determine if the ranger noticed the threat. Being a Dungeon Master requires making a lot of these subjective decisions, and when there’s debate I always encourage looking for an objective, mechanical way to settle debates like an ability check.
“is surprised at the start of the encounter” is mostly simple, but does have one strange implication: if a creature joins the encounter part-way through, as the rules are written they are not surprised. Even if the fight literally falls through the ceiling above a creature, they’re somehow not surprised. I typically recommend bending the rules here, but we’ll cover joining an ongoing encounter in the Initiative section later in this guide.