I'm not sure if there's one single "certain ability" that makes students able to solve this problem. In order to solve this problem, a student should:
- Know what it means to "define an operation between integers."
- Know what an equation like $a * b = a b - a + b$ means when it's interpreted as a definition.
- Recognize that it is both permissible and useful to apply the substitution property to this situation.
- Know how to apply the substitution property to this situation.
- Recognize that, after the substitution property has been applied, the answer to the new problem is the answer to the original problem.
These are all distinct pieces of knowledge, and it's possible for a student to possess any four of these but not possess the fifth.
That said, these pieces of knowledge are all fundamental facts about elementary algebra. (Of course, "fundamental" doesn't mean "easy to learn"!) So I think I'd summarize these as something like "a solid understanding of the fundamentals of elementary algebra."
In response to your comment:
If you were to put all the 5 bullet points in a box and give it a label, what would you write on that label? "A solid understanding of the fundamentals of elementary algebra" doesn't seem to capture the concept, because as far as I know, in elementary algebra they don't teach equations like $4∗x=36$.
That's a great question, and I'm not really sure.
Let me start by clarifying that when I say "elementary algebra," I mean "manipulating expressions and equations involving numbers, and especially solving equations"—in other words, the area of math that's usually just called "algebra" in the context of secondary education.
Next, let me pretend for a moment that the problem that you wrote was:
Define a function $f$ on integers as follows: $f(a, b) = a b − a + b$. Solve the equation $f(4, x) = 36$.
Now that I think about it, I think that there is one particular skill that a student needs in order to solve this problem. If I had to give a name to that skill, I think I would call it familiarity with functions. This skill consists essentially of the pieces of knowledge I listed above: knowing what it means to define a function; knowing what an equation means when it's interpreted as a function definition; and knowing how to use the substitution property with a function definition.
In order to solve the problem you originally described, a student needs to have familiarity with functions, and also needs to know that "define an operation $a * b$" means exactly the same thing as "define a function $f(a, b)$" (aside from the difference in notation).
(If I may ramble for a few moments, I think that there are two important "levels" of familiarity with functions. The first level is what I described above—it's the level where a student is capable of reading "$f(4, x) = 36$" as meaning "$a = 4$, $b = x$, $y = 36$." The second level is where a student is capable of treating a function as an independent object in its own right, and solving problems that involve multiple values of the same function without getting confused.)