There are lots of ways that rules, not just written rules, can change your game. I wrote a big list of ways they can do this.
But rules only work if you enact them. No rule can make you do anything, make your group do anything — you have to make rules work by following them.
This sounds obvious, when you say it like that, but looking at arguments online it’s clearly not obvious to everybody. I’m not sure it was always obvious to me.
Question — but don’t people generally follow the rules in rpgs?
No. People routinely ignore and customise rpg rules. Often for good reasons, but also often because they don’t notice, or don’t realise what they’re losing by doing so.
Objection — rules aren’t the only thing that matter
No. Of course not. Personalities, relationships, playing environment, skill and experience all matter.
(There was a trend at one time to make hierarchies — “people, environment, snacks, system, in that order”. Those are silly — you can’t pull a complex system apart like that. A good starting point is to think about bottlenecks or weakest points — a serious problem with any one of rules, personalities or play environment can make a mess of things.)
Objection — most games have basically the same rules
Yes. Rules can change your game, in interesting and desirable ways, but many rules don’t do that. Justin Alexander gave us good wording for this — “most RPG systems don’t actually carry a lot of weight, and are largely indistinguishable from each other in terms of the type of weight they carry”. Most trad rpgs have the same very lightweight framework around major game events and “plot”-significant elements; to a large extent, they say “the GM decides”. They also have huge fine-detail subsystems, but those don’t fundamentally change how the game plays out — they don’t change the kind of larger scale narratives that it produces.
Once you step outside the trad-rpg circle, though, there are a host of games that can radically change the way you play.