I was put in charge of assessment for our community-college math department as of last year. One resource that was immensely useful was the UMass "Program-Based Review and Assessment" handbook (link). There's an example of a math department assessment plan on p. 28.
That said, the process highlighted there is exactly as you describe: score selected questions embedded on final exams. On the one hand, we're fortunate in the math discipline that assessment is so amenable to our existing testing practices; on the other, it does look a bit silly and redundant. The silver lining for me is to know about specific skill mastery and not overall test or course averages.
The one thing that mystifies me is the practice of checking only one objective per year; that doesn't seem like the intent to me (although the UMass document, for example, does say "if you and other members of your department can agree on only one goal, don’t let this stall your progress. Focus on that one goal – more will come later." [p. 11]). Assessing just one goal per year seems like you'd be going a half-decade between pings on any objective to know whether they'd degraded or improved or whatnot, which seems crazy.
What I'm doing is to establish fixed questions in each of our terminal courses, and so assess every one of our objectives every year in a consistent manner. Hopefully this actually leads to less effort, because we won't have to spend mental energy creating new plans (re-inventing the wheel) every year; will also provide a "dashboard" of information on critical skills to compare year-to-year; and serves as a resolution to your very valid "one exam question can be misleading and not representative of an entire program" issue. Note also this is just the terminal courses (last ones to complete the degree), which I think meets the spirit of program assessment (what do students know when they complete our time with us), and also conveniently has a relatively smaller number of sections, and students, and instructors to wrangle for the task.
As extra steps I'm also receiving prerequisite grades and academic majors for all our students (to perform cross-correlations from the prerequisite course to identify particular strength or weakness), and sending out a survey to alumni this month for observations, how they did afterward, things they wish they were improved, and perceived strength on each of our program objectives. But that's definitely extra-credit for this task (I was expressly told by our Dean of assessment not to do that because it would be too much work), and can certainly be skipped if you just want to minimally meet accreditation demands. However, I did want to add a bit more depth/texture to our understanding of how our students are doing in later academic programs.