Series (2 of n): How practice questions work

This series was introduced in a post on August 5th.

This installment address the approach to creating the questions. Reverse engineer the process; the best way to understand technical exams is to try to write one yourself. Keep in mind the following criteria:

You want about a 65% average score the first time they take it, assuming an appropriate audience. Too easy a test is a waste of their time and to difficult a test is a transparent display of how much you think you know or can look up on Google. The practice exam is a teaching tool, first and foremost.

Now, consider this simple approach to just one individual question:

• What do you want the tester to prove he understands?
• Is this better asked directly or indirectly?
• Should they answer the right answer or illiminate the from the wrong ones?
• Is this a question where distractin noise is appropriate, or should you just keep it short?
• What objective of the final "real exam" goal is this practice preaparing them for?

Every practice question can take from 10-30 minutes to create from concept to explaination. In a business day then it would be production to crank out 30 questions. The real questions might have hours of argument from a board of brains behind them. These aren't just made up random trivia, each must be thought through. Each question and false choice has a purpose.

Now, as you are studying for your next exam....try to anticipate what really seems to capture the truth and presence of the class. Step into the shoes of the psycho(metrician) and ask what would it take to fool...you.

Process Oriented Programming

Often times in the CISSP and Security+ classes we are confronted with the need to come up with examples that illustrate detailed terms that don't translate well into "business language".

Some things just suck if they were to happen. And explaining this to a cost/benefit manager is sometimes an exercise in awkwardness for both parties. Here is a good example for the "programmery" (my term) knowledge domains in CISSP. The ones where we get into the weeds about registers and processes and so on:

Follow this link http://www.physorg.com/news169133727.html

This is a practical example of injecting instructions to a process while it is running, voting machines make an example everyone, not just those that work on the secret systems most will never see can understand.

"How to solve it"

While researching a test question about virus scanners I ran across these ideas regarding "heuristics". It came from a book written in 1945 called "How to solve it".

These are probably the best four suggestions I can give a student on how to deal with the CEH/ECSA/LPT materials. Remembering first off that perhaps the most fundamental heuristic is "trial and error".

1. If you are having difficulty understanding a problem, try drawing a picture.
2. If you can't find a solution, try assuming that you have a solution and seeing what you can derive from that ("working backward").
3. If the problem is abstract, try examining a concrete example.
4. Try solving a more general problem first (the "inventor's paradox": the more ambitious plan may have more chances of success).

Series (1 of n): Using practice exams effectively

Part of the current book project I am doing involves writing practice questions. In doing this I have put a lot of thought into the topic and wanted to share some of that here.

First, just to get the controversial part out of the way, I believe in practice questions. They are ethical and it is fair to try to get them as close to the real thing as possible, at least in terms of scope, style, and difficulty level of the real test. That is my opinion and other instructors might disagree.

A risk of providing practice exams is realized if the student can subconsciously understand them to mean "The instructor is essentially taking this test for me, if I do what these questions say and I will pass." I say subconsciously because I have never heard a student actually say this out loud, but I can tell by the way they ask questions about the exam and their general preparation habits when this perception is taking hold. This is the source of the understandable criticism of practice tests, but it can be managed and handled correctly.

As I write the questions for the book, I am placing in some controls. In the interest of security-open-source-minded full disclosure I don't mind explaining them. The best cryptosystems are well known and understood, but are still hard to solve. That is the good model for practice exams as well. Along the way, discussions about real exams are likely to be brought up as well.

To keep the blog postings reasonable in size, I will address specific topics of practice exams, and how to get the most out of them over the course of several postings. In case you are working on some right now start with this thought:

“Practice exams are extensions of lab, lecture and other learning modes. Not replacements for them, and not shortcuts to avoid them.”