Status: fairly confident in this now
Why bother with rules for roleplaying games? Whether RAW from a third party text, hand-crafted by the GM, or assembled by the play group through a democratic process, rules require effort to understand, require effort to remember in play, and may give weird, unwanted results at times.
So why not just freeform, using group consensus of equals or by appointing a GM and respecting their judgement? After all, freeform games are easier to set up and more flexible in play. And if you have a stable group, or a stable play circle, you can hone your freeform play with informal procedures and conventions that meet precisely your needs. Why try to impose formal rules on top of that?
In other words, what do rules ever do for us?
Rules can define the roles of the players
Rules can explicitly distribute agency. For clear examples in GM’d games, look at how Dramasystem or Inspectres limits the GM’s powers. In GMless games, this is a very large part of what rules do. Look at how Archipelago (free download) distributes control of elements of the fiction (e.g. “magic”, “geography”, “history”), mostly protects your control of your character, and establishes how your character control can be taken by others:
You control your character’s actions, thoughts and dialogue at all times. […] Nobody can say Try A Different Way when you narrate about your character.
… The only exception is when someone tells you That Might Not Be Quite So Easy – then whoever [randomly] gets to narrate the outcome of the situation can narrate what your character does, and what happens to them. …
… If someone narrates something that you disagree with within an element [of the fiction] you own, you can veto their narration. This is something you can do for example if the narration contradicts existing knowledge, if it dilutes your vision of the element, or if it changes the element in a way you don’t think works. …
Rules can distribute agency even in the presence of a strong GM, albeit to to more a limited degree. E.g. if we are playing “D&D 5e”, I’d expect that the movement, range, and action economy rules hold whenever we’re in combat. This allows me to have some confidence about my tactical decisions. 
Within a given agency distribution, rules can resolve disputes about the possibility and difficulty of a given action. On Reddit, /u/ParameciaAntic illustrates this —
For example, if I’m playing Johnny Deadshot, fastest gun in the West, then I might think that he could easily shoot the three charging bandidos off their horses. The GM may think that due to the distance, terrain, and rapid speed that he might only be able to hit one.
The rules provide a set sequence of procedures to follow in order to resolve this situation – probability to hit based on skill and factors like terrain, number of shots allowable within the time frame, damage from the weapon, etc.
Generalising the previous point — they can make it easier for a GM to be fair, and to convince others that they are being fair. This aids what Eero Tuovinen describes as the “hygienic practice” that helps GMs build player trust.
Rules can shape the events of play
Rules can provoke players to say interesting things. If nothing else, rules can push players towards actions they might not have considered, perhaps by having rules about them, perhaps by having rules that enable them (such as by superhuman powers). PBtA games, which provide playbooks full of player moves, seem particularly good for this.
(I attribute the above to Vincent Baker originally, but a brief search didn’t come up with a source for it)
As a specialisation of the the above, rules can cause players and GMs to do unpleasant things to PCs and cherished NPCs. In particular, they can help the GM cause real damage to PCs, to kill PCs, rather than balking and giving them an easy out. Or to say that, yes, they do have an old army friend, but she is secretly in the pay of their enemies (think of the Enmity Clause in Burning Wheel). Vincent Baker talks about this in Rules vs Vigorous Creative Agreement —
“… what you want are outcomes that upset every single person at the table. You want things that if you hadn’t agreed to abide by the rules’ results, you would reject.”
More broadly, rules can guide player focus, and particularly player time, towards particular aspects of play. By agreeing to follow certain explicit rules, and actually doing so consistently (not a given, of course), we shift our focus to the things that those rules cover. As Manixur says on Story Games —
“In freeform roleplay […] even if we come to a detailed explicit agreement about why and how we’re playing, we’re at risk of diverging from that plan or falling into our habitual modes of interaction in the absence of some kind of third-party reinforcement.”
Rules can speed up play by providing structure and routine, reducing the need to continuously negotiate. (thanks to Vivificient on Story Games for this observation)
As well as shaping play directly, rules can provide feedback on whether you’re meeting your play goals. For example:
- Are PC Beliefs in Burning Wheel working for your group? I.e. are the players pursuing them aggressively and is the GM enabling that pursuit? You can tell by looking at the Fate and Persona points awarded for them each session. Few or no awards related to Beliefs is warning sign that they’re not working.
- Is your Dungeon World game about exciting fantasy adventure? Look at the three XP questions in the “End of Session” move . If you’re not awarding many XP for them, then maybe your game is not about that.
- Are your Blades in the Dark PCs really acting as Smugglers? Or would some other crew type model them better? Look at the crew XP question that you answer each session. 
Rules can help manage genre, setting, or tone
Rules can help manage important thresholds, such as that between life and death, so that the risk is at the right level. Consider Burning Wheel for example — it’s a game about consequences of actions, but it’s all about PC goals, so PC death is extremely disruptive. So the combat rules make it easy to be seriously hurt but very hard to die.
Rules can teach setting. Burning Wheel does this particularly well — to create a character you must choose a course through several lifepaths, obeying many specific restrictions on what lifepath can lead to what, and many lifepath-specific rules on what mandatory skills and traits you must take. The work of this search will burn into your brain many details of the implicit setting. (We learn much better when we work hard to use material than we do when we mere read it.)
Rules can be their own reward
Rules can provide a tactical challenge, something to master to achieve your goals. This is great source of pleasure for many players. Doing well at it (achieving rules mastery) is also a way to signal intelligence (and general competence at getting systems to do what you want).
 Thanks to various commenters in the rpg.net thread, starting with dbm in post #2, for bringing “rules distribute agency” to my attention. Even though it’s a familiar idea (e.g. it was a central at the Forge), I’d originally ignored because I was thinking most about strong GM games, and the kind of agency in the paragraph above doesn’t ensure I will have agency in the things I’m really interested in — control over narrative and plot. In a game with a strong GM role, those are essentially granted by the GM, ultimately following the results of metagame negotiations. As a player in a GM’d game, I prefer having very few formal rules, because it lets me focus on that negotiation (or the GM’s unwillingness to engage in it).
 “… Then answer these three questions as a group:
- Did we learn something new and important about the world?
- Did we overcome a notable monster or enemy?
- Did we loot a memorable treasure?
For each “yes” answer everyone marks XP.”
 E.g. for Smugglers — “[Did you] execute a successful smuggling or acquire new clients or contraband sources.”