One thing that seduced many players away from D&D back in the day (at least in my experience) was the combat system. I used to frigging HATE AD&D‘s combat system. 1 minute rounds? d4+1 damage for a crossbow? Mid-level characters with 50 or more hit points, so that they take 10 or more sword wounds to kill?
“Don’t worry, I still have 6 hit points left!”
Of course I was occasionally persuaded that the hit points were just abstractions, as were the combat rounds and damage dice. It’s only a game, after all, right? The DMG explains at some length that one minute rounds assume lots of parrying, dodging, etc. with one real opportunity to get in a hit. But then weapons specialization, various stunts and maneuvers, and so on were added, each incrementally reducing the abstraction for me.
Rolemaster was the first serious alternative to AD&D that I remember playing. That game had higher hit point totals, but assumed, IIRC, that they represented actual damage (the skill “Body development” was the only way to gain hit points) and because a character had 30 or more h.p., bleeding could be assessed at a few points per round, and so on. Most importantly, a critical hit system made many, if not most, successful hits by weapons deadly. Many a Rolemaster character bled out from a severed femoral artery, or suffered shattered bones from a mace, as well as amputations, permanent nerve damage, and pretty much anything that could “really” happen when crazed orcs and dwarfs are swinging axes and swords. A round was 6 or 10 seconds, I think, and armor actually made you harder to wound, not harder to hit. Good stuff but each frikking weapon had a separate combat matrix to roll on! (A falchion was one of my favorite weapons, just based on appearance and theory, and I was really glad to see that in RM they were among the best.) When we played RM,we had to assign one player the task of keeping track of the critical hit tables, another tracked the convoluted XP system, and another mapped. It was a lot more like homework than gaming after a while.
A bit later my brother & I found GURPS, beginning with the first edition, and it had 1 second (!) rounds. Weapon damage was closely tied to the wielder’s strength and whether the weapon chopped, crushed, or pierced the victim. With one-second rounds, unbalanced weapons (flails, axes, etc.) generally took a full second to recover after a swing, while swords did not, and while swords generally did a bit less damage than the hafted weapons, they were FAST. The mechanics seemed to effectively lead to realistic choices by players (yes, I’ll have all the armor I can carry if we’re going into battle, and give me a sword unless my opponent is very heavily armored, etc.). Shields were much more important than they had been in AD&D (a flat 5% bonus to defense) or Rolemaster (a 15 or 20 point bonus in a system where a percentile roll is modified by potentially huge numbers). In GURPS a shield can actively be used to block a blow, and passively may deflect a blow just by being in the way.
So GURPS won the realism laurel, and we really didn’t try anything new until 3rd , 3.5, and 4th edition D&D came about. All were more tactical than AD&D or Rolemaster (and about equal to GURPS) but they retained almost all the crazy artifacts of the abstract AD&D system, grafted onto a tactical movement system that only drew more attention to how unrealistic everything else is.
So I’ve actually come full circle, preferring the abstract combat of ODD or AD&D, even entertaining ODD‘s “all attacks do d6 damage” should I DM my own campaign. Right now we play C&C and I am more than happy.
It turns out that realistic combat has the excellent advantage of making players more careful about when to fight, but also tends to make character death an almost unavoidable consequence of a few bad rolls. More importantly, although I’ve played many serious gamers who love the challenge of mastering complex rules, no one I game with has the time for that, myself included. Let me roll a d20 and be done with it, there’s a whole dungeon to explore!
(YMMV. For settings outside the D&D milieu, like Old West, Sci-Fi, Horror, and so on, more realistic and deadly combat systems have a place. I can’t really imagine playing swashbuckling type games in anything but GURPS, for example.)
You should try Melee for a realistic and quick combat system. It was created by Steve Jackson, sort of a forerunner of Gurps. The system was called The Fantasy Trip or TFT and consisted of Melee (the combat system), Wizard (Magic system) and In the Labyrinth (DMs guide). It went out of print back in the early 80’s but you might find a copy on Ebay or Amazon
I’ve always thought TFT looks like a great game. I wonder if Melee could be grafted onto D&D or Labyrinth Lord as the combat system…
I think my brother actually owns Melee and I’ve looked at pdfs of the rest of the rules online; there was a guy in Texas who posted scans of everything on his web site, which of course is probably illegal; I don’t know if his site still exists.