I spent some time looking for information that might provide some kind of reliable, evidence-based answer to my own question. I came up with the following, which is not perfect, but I thought it would be worth posting as an answer. I tried two methods of probing this: standardized test scores of students with high socioeconomic status (SES), and heritability of standardized test scores as measured by twin studies.
Standardized test scores of students with high SES
For this method I used data from the US.
We can't use success in high school algebra as a way of determining innate ability to learn algebra, for a variety of reasons. Students in some states are required to take algebra as early as 8th grade, but many are not developmentally ready at that age, so their failure doesn't necessarily indicate a permanent inability. Anecdotally, many US high school algebra teachers report that they pass students who haven't learned the material but who were punctual, obedient, and turned in homework. Because many high schools have algebra as a graduation requirement, there is intense political pressure to say that students have passed algebra, regardless of whether they really understand it. Many students fail algebra not because of a lack of innate ability but because of a lack of motivation or interest. Many public schools serving a student population with low SES do not provide competent instruction.
To get around these problems, I looked for data on SAT math scores among kids with high SES. Among the most affluent families in the US (the top decile of SES), about 80% of kids go to college (although, surprisingly, only about 2/3 of those get a degree). The vast majority of these kids were exposed to competent instruction in algebra during high school, and they were typically under intense social and family pressure to get into a good college. Nearly all of them took the SAT, which is required by almost all schools in the US that have selective admissions.
For students with the highest SES (more than $200,000 per year of family income), the average SAT math score is about 600. A score of about 600 is also used as a cut-off by many colleges in determining which of their students are allowed to take calculus without having to take a prerequisite course. The math section of the SAT includes a lot of algebra. Therefore it seems reasonable to me to take a score of about 600 as indicating pretty decent mastery of algebra. If the mean of the scores for this group roughly coincides with the median, then about 50% of people in this group have mastered algebra.
It seems reasonable to assume that there is no significant difference in innate mathematical ability between the most affluent people in the US and the rest of the population. Therefore the scores of this group can probably be interpreted as a rough measure of what level of achievement you get in math, at this age, given a favorable environment. That is, they may give a reasonable estimate of what nature permits, given the right nurture.
The mean score for Asians (regardless of SES) is also about 600. If we assume that there is no innate difference in mathematical ability between Asians and other ethnic groups, then this would seem to support the hypothesis that a mean score of about 600 represents a broad average of potential mathematical achievement, among college-bound kids at age 18, and that this potential is reached if these students receive competent instruction and grow up with cultural expectations that pressure them to do well in academics.
Since about 80% of high-SES students go to college, and about 50% of those get an SAT math score of 600 or higher, my estimate would be that something like 50-60% of high school kids in general are not capable of learning algebra. This seems to be in the right ballpark to support Daniel Collins's statement that:
Most of our students will never grok 8th-grade algebra no matter how many times they try, nor under what circumstances.
Many people who don't master algebra in high school nevertheless go to college, where they will typically be required to take remedial ("developmental") courses in math. Some succeed in learning algebra at that point, so we should correspondingly reduce our estimate of the percentage of the population unable to learn algebra. However, remediation in math appears to be extraordinarily ineffective.
The most questionable assumption in my estimate seems to me to be that high-SES kids who get less than 600 really tried to learn math in high school. They were probably under strong pressure to succeed in math, but they may have responded to that pressure by doing the minimum required in order to get a certain grade, and doing the minimum may have meant going through the motions of taking a math course without actually learning the math. If there were many students like this, then the percentage of the population lacking the innate ability to learn algebra may be much lower than the above estimate. The size of this bias is not likely to be huge, however, because then one would expect a significant number of these students to succeed when remediated in college, whereas in fact remediation almost never works.
Since there are a lot of shaky assumptions in the above estimate, it's interesting to see if we can place any more definitive upper and lower bounds on the figure.
As an upper bound, the percentage of high school kids in the US incapable of learning algebra must be no more than about 94%. This can be inferred because out of 4.3 M Americans aged 18, about 1 M take the SAT, and 25% score 590 or higher on the math portion. Thus about 6% of the population demonstrates mastery of algebra every year, when they reach this age.
As a lower bound, the percentage incapable of learning algebra seems sure to be at least 3%, since that's roughly the percentage of the population that is considered intellectually disabled.
Heritability of standardized test scores as measured by twin studies
Kids in the UK have to go to school until they're 16, and at that age they take a set of standardized exams called GCSEs, which includes a test of math. The Mathematics GCSE's "higher tier" is 30% based on algebra. The level of the algebra questions is about at the level of a first high-school algebra course in the US, i.e., it overlaps with the math portion of the SAT, but doesn't include the most difficult material you would find on the SAT.
Twin studies show that GCSE math scores are 55% heritable. Only about half of this genetic influence appears to be due to intelligence; the other half seems to be because of other partially heritable personality traits such as "self-efficacy" and behavior problems.
I would like to study this approach more thoroughly, but these findings seem to show that we can't necessarily think of a hypothetical innate inability to learn algebra solely as an innate intellectual shortcoming. For example, a student who refuses to do his algebra homework may be doing so partly because the heritable component of his intelligence tends to make the activity hard and not enjoyable for him, but also partly because a partially heritable component of his personality makes him unwilling to stay on task in order to achieve a future goal.
If the twin studies obtained data on SES, then it would be interesting to combine the two approaches.
A counterintuitive part of the interpretation of these data is that high heritability of test scores goes hand in hand with a more socially equal, meritocratic society. Suppose we make some change in social policy that makes us more meritocratic, e.g., Head Start programs, or eliminating the practice of giving preferences for college admission to children of alumni. By doing this, we've reduced the amount of variability in the environmental factors affecting education. Therefore the amount of variation in educational achievement will go down. Of this smaller remaining amount of variation, the fixed amount due to genetics will now represent a greater fraction.