My own rules of data analysis
Comments (0) The answer you get depends on the question you ask
In many substantive fields, students take one, two, or perhaps three statistical courses during graduate school. These typically cover things such as descriptive statistics, ANOVA, regression, and perhaps a couple variants of regression such as logistic regression. These are good tools for many purposes, but it’s a very limited toolbox. This limits the number of questions you can ask. Perhaps the really interesting substantive question is one that you can’t answer with those methods. But if you ask a statistician or data analyst, you may find that the right method to answer your question does exist.
You can’t see something you’re not looking for
The more specific your question, the better you will be able to answer it; but if it’s too specific, you may miss something else. Researchers need to learn to adapt the focus of their investigations.
If you’re not surprised, you haven’t learned anything (well, not much, anyway)
Isaac Asimov once said “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ (I found it!) but ‘That’s funny …’”. That is, surprising. It’s fine to confirm what you already suspected, but the real advances are made when you find things you did not expect.
and
Any analysis worth doing can be done in more than one way
This gets back to the toolbox – Which method should I use? but, even within a method, there are often options. Should I transform variables? Which covariates should I include? How complex should my model be? What effect sizes should I report?
Often, these and other related questions do not have simple answers, but rather a range of reasonable choices.
