Thursday, October 22, 2009
Lecture 5 : R&D methods
GOALS OF R&D (Research and Development):
- integration of theory and practice
- create something novel, new
- suitable to be used widely in computing field
TYPES OF R&D METHODS:
- action goals - develop technology to solve social problem
- development goals - solve social problem while simultaneously constructing generalizable design principles
- constructive goals - delevop innovations and novel technologies
- integration of theory and practice
- create something novel, new
- suitable to be used widely in computing field
TYPES OF R&D METHODS:
- Action research - "how to" question, not "why" and "what"; link theory with practice; thinking with dogin; observer is a participant - participative; develop product or intervention to solve practical problem; advancing local knowledge but not neccessary attempt to construct generalizable theory or pronciples; evolving product over several cycles; iterative; research in natural settings;
- Development research - create artifact; develop a solution to real problem or evaluate and refine theories; problem-oriented; design experiments; in ICT similar to constructive research; iterative; refine; in situations requiring practical and research contributions;
- Design research - create artifact; solve real-world problems; produce theoretical outputs; implement IT soutions in authentic settings; contextual; cycles; teamwork; data from multiple sources;
- Constructive research - origin in computing field; close to action and design research; innovative constructions; in the real world; construct a solution and show that it works, examine scope of applicability
Lecture 4 - qualitative research methods and data collection methods
QUALITATIVE RESEARCH METHODS
1. Enhomethodology
- originated from social sciences, answers "how a certain community behaves and how the principles and values of the community affect the behaviour of the inviduals", investigate aspect of cultural communities: norms, values, pratices; target at specific population; investingates whole community, not individuals;
"+" holisic, investigates complex phenomena
"-" lot of time, high skills, validity of the reports, if it is science at all
2. Narrative reserach
- study of single individual and their life experiences; they provide stories about their life experiences; following things should be addressed: phase of life, single theme, specific situations; not only biographies are sources of data, also interviews, field notes, observations, storytelling, newspapers; can be applied in many fields;
3. Phenomenography
- investigate inviduals' level of understanding an experience or phenomena; usually by interview about this experience or phoenomenon; used mainly for investingating learning process, understanding of learned content, but also for desctibing th conceptions of the world around us;
4. Case study
- detailed analysis of a signle object or phenomena (person, system, organization, course, ...); real life context; "how" and "why" questions; to investigate borders between phenomena and context; in natural settings; hard to generalize; concrete knowledge may be considered less valuable; most for generating hypotheses; we can compare between cases afterwards;
5. Grounded theory
- inductive; create theory from data; collect data, select data, and analyse data; indenfity categories in data, build relationships between them, group together;
6. Surveys (also)
DATA COLLECTION METHODS IN QUALITATIVE AND MIXED RESEARCH
1. Enhomethodology
- originated from social sciences, answers "how a certain community behaves and how the principles and values of the community affect the behaviour of the inviduals", investigate aspect of cultural communities: norms, values, pratices; target at specific population; investingates whole community, not individuals;
"+" holisic, investigates complex phenomena
"-" lot of time, high skills, validity of the reports, if it is science at all
2. Narrative reserach
- study of single individual and their life experiences; they provide stories about their life experiences; following things should be addressed: phase of life, single theme, specific situations; not only biographies are sources of data, also interviews, field notes, observations, storytelling, newspapers; can be applied in many fields;
3. Phenomenography
- investigate inviduals' level of understanding an experience or phenomena; usually by interview about this experience or phoenomenon; used mainly for investingating learning process, understanding of learned content, but also for desctibing th conceptions of the world around us;
4. Case study
- detailed analysis of a signle object or phenomena (person, system, organization, course, ...); real life context; "how" and "why" questions; to investigate borders between phenomena and context; in natural settings; hard to generalize; concrete knowledge may be considered less valuable; most for generating hypotheses; we can compare between cases afterwards;
5. Grounded theory
- inductive; create theory from data; collect data, select data, and analyse data; indenfity categories in data, build relationships between them, group together;
6. Surveys (also)
DATA COLLECTION METHODS IN QUALITATIVE AND MIXED RESEARCH
- Interview - open-ended (unstructured), closed (structured, data better for comparison), or focus group interviews (discussions in groups 8-12 people, usually recorded); do not show your opinions to respondedts;
- Direct observation - non-intrusive; can be recorded by videa camera; could be free form, or eg. counting how many times a behaviour occures, or mixed; observed things are: participants, interactions, nonverbal behaviour, physical surroundings; use in requirement capturing, evaluation, ..; interviews can be conucted later to clear something; ethically we should inform people they're being observed;
- Log files
- Content analysis - study the text content; widely applied; eg. to study authorship, authencity, meaning of communication; we create some code book but no time to think about it..
Lecture 3 - qualitative research
QUALITATIVE RESEARCH
key words: study context, subjective findings, time- and context-dependent, inductive, experiences of people, "how" and "why" questions, naturalistic settings, purposefully selecting participants, study of invidual cases;
*triangulation - collection of data from many sources
Data sampling - purposive, extreme situations, maximize variation, or snowball, or random
Data analysis - data reduction, data display (tables, charts), conclusion
Validity - asses if the researcher is objective, if the results can be generalized beyond the research context, if research is repeteatable, scope of data, depth and richness of data
Reliability - if the same results would be made in different place or time, if the same observations would be made if the researcher had paid attention to different phenomena, eg. we can use two researches to gain that;
watch out for: holistic fallacy - tend to interpret data as more patterned and congruent that in real; elite bias - high-status informants overweight the data; going positive
Phenomenology - eg. study phenomenon from students' point of view
Constructivist - eg. how students themselves construct models of what the learn
MIXED RESEARCH
- use both qualitative and quantitative research methods;
Advantages: (lanie wodyyyyy)
+ overcome weakness of single method study
+ get more holistic picture
+ numbers can add precision
+ pictures add meaning to numbers
+ provides stronger evidence
key words: study context, subjective findings, time- and context-dependent, inductive, experiences of people, "how" and "why" questions, naturalistic settings, purposefully selecting participants, study of invidual cases;
*triangulation - collection of data from many sources
Data sampling - purposive, extreme situations, maximize variation, or snowball, or random
Data analysis - data reduction, data display (tables, charts), conclusion
Validity - asses if the researcher is objective, if the results can be generalized beyond the research context, if research is repeteatable, scope of data, depth and richness of data
Reliability - if the same results would be made in different place or time, if the same observations would be made if the researcher had paid attention to different phenomena, eg. we can use two researches to gain that;
watch out for: holistic fallacy - tend to interpret data as more patterned and congruent that in real; elite bias - high-status informants overweight the data; going positive
Phenomenology - eg. study phenomenon from students' point of view
Constructivist - eg. how students themselves construct models of what the learn
MIXED RESEARCH
- use both qualitative and quantitative research methods;
Advantages: (lanie wodyyyyy)
+ overcome weakness of single method study
+ get more holistic picture
+ numbers can add precision
+ pictures add meaning to numbers
+ provides stronger evidence
Lecture 2 - quantitative research methods
QUANTITATIVE RESEARCH METHODS
1. ~Survey
- set goal - what you want to learn (why, what or how); watch out that correlation does not imply casuality (reason-cause relation);
- sample - determine who will be interviewed; the bigger sample the better; try to avoid bias (nice people, customer people, older people, ..);
- methodology - how you will interview: personally, by telephone, by mail, by email, as online survey (each has different risks of bias or other disadvantages);
- questionarre - create it; useful rules:
- collect and analyse results
- communicate results
2. Controlled experiment
- in general: construct some cause-effect relation, then apply some treatment and make observations;
HYPOTHESIS
Every study must have at least 2 hypotheses (because they must cover all possibilities):
H0 - the treatment has no predicted effect
H1 - the treatment has the predicted effec
Types of hypotheses:
We should have expetimental group of one or several levels of treatment plus control group against which we will compare the treatments, the treatment there can also be "no treatment".
STATISTICAL METHODS
Types of variables:
** TODO
3. Mathematical modelling and simulation
4. Theorem proving
5. ~Participant observations
6. ~Grounded theory
1. ~Survey
- set goal - what you want to learn (why, what or how); watch out that correlation does not imply casuality (reason-cause relation);
- sample - determine who will be interviewed; the bigger sample the better; try to avoid bias (nice people, customer people, older people, ..);
- methodology - how you will interview: personally, by telephone, by mail, by email, as online survey (each has different risks of bias or other disadvantages);
- questionarre - create it; useful rules:
| ----> | * start with introduction or welcome message, then explain the reason for the survey (shortly); why it is worth taking it, how it will influence our lives, etc. * reassure confidentiality * instructions at the point that needed, not earlier * early questions should be easy and not too personal * avoid technical terms * change question types not to make people not think about what they are doing * keep it short * PRETEST! QUESTION TYPES: - open-ended - closed-ended * include "none" and "other" * include most possible choices to avoid bias - Likert scale * start from "Strongly agree" - rating scales * 1..10 or 1..5; or 4..10 - in Finland * higer numbers more positive * include "non applicable" |
- collect and analyse results
- communicate results
2. Controlled experiment
- in general: construct some cause-effect relation, then apply some treatment and make observations;
HYPOTHESIS
Every study must have at least 2 hypotheses (because they must cover all possibilities):
H0 - the treatment has no predicted effect
H1 - the treatment has the predicted effec
Types of hypotheses:
- One-tailed - specifies direction of the difference
- Two-tailed - does not specify direction of the difference
We should have expetimental group of one or several levels of treatment plus control group against which we will compare the treatments, the treatment there can also be "no treatment".
- Between subjects design - each subject exposed to one treatment level; can be done with pre-test - measure the dependent variable also before the treatment;
- Within subjects design - each subject exposed to more than 1 level of experimental treatment; but has some drawbacks, eg fatigue effect, learning effect;
STATISTICAL METHODS
Types of variables:
- nominal, eg. male/female, Java/Python/C;
- ordinal, eg. Likert scale (but intervals may be uneven);
- interval, eg.Celcius temperature;
- ratio, eg. Kelvin temperature (there's logical 0);
- central tendency - mean, median (middle value), mode (most frequently occuring)
- dispersion - range (=max-min), variance, stardard deviation (sqrt(variance));
- distribution - frequency, percentiles, charts, plots
** TODO
3. Mathematical modelling and simulation
4. Theorem proving
5. ~Participant observations
6. ~Grounded theory
Lecture 1 - general info
STEPS OF SCIENTIFIC METHOD:
ad. 4 (experiment) - everything should be described in detail, will be run for different groups; each experiment should have control group (exposed to same conditions except for variable being tested);
ad. 5 (results) - tabulate, draw figures, use statistical testing, confirm by retesting if possible
ad 6 (conclusion) - accept or reject the hypothesis
RESEARCH PARADIGMS IN CS:
- aim to classify count fatures, construct statistical models
- researcher knows in advance what they're looking for
- recommended in latter phases of research
- all researh is carefully designed beforehand
- data is numbers and statistics
- objective, as uses tools eg surveys
- result more likely to be generalizable
QUALITATIVE:
- aim is complete detailed description
- researcher knows only roughly what they're looking for
- recommended in earlier phases of research
- the design of research emerges as the study unfolds
- data is in the form of words, pictures, objects; rich data
- subjective, researcher interprets
- the result is less generalizable
CONSTRUCTIVE RESEARCH PHASES:
- finds problem
- examine if some research could be done
- start to gather better undestanding of reserach area
- cyclically come up with solution (first design, then implement) and test it
- think about the scope of applicability of the solution
- analyze theoretical contribution
DEDUCTIVE REASONING order:
- Problem / question
- Observation /research
- Formulate hypothesis
- Experiment
- Collect and analyze results
- Conclusion
- Communicate results
ad. 4 (experiment) - everything should be described in detail, will be run for different groups; each experiment should have control group (exposed to same conditions except for variable being tested);
ad. 5 (results) - tabulate, draw figures, use statistical testing, confirm by retesting if possible
ad 6 (conclusion) - accept or reject the hypothesis
RESEARCH PARADIGMS IN CS:
- Quantitative (positivism)
- Qualitative (interpretivism)
- Development / constructive
- aim to classify count fatures, construct statistical models
- researcher knows in advance what they're looking for
- recommended in latter phases of research
- all researh is carefully designed beforehand
- data is numbers and statistics
- objective, as uses tools eg surveys
- result more likely to be generalizable
QUALITATIVE:
- aim is complete detailed description
- researcher knows only roughly what they're looking for
- recommended in earlier phases of research
- the design of research emerges as the study unfolds
- data is in the form of words, pictures, objects; rich data
- subjective, researcher interprets
- the result is less generalizable
CONSTRUCTIVE RESEARCH PHASES:
- finds problem
- examine if some research could be done
- start to gather better undestanding of reserach area
- cyclically come up with solution (first design, then implement) and test it
- think about the scope of applicability of the solution
- analyze theoretical contribution
DEDUCTIVE REASONING order:
- Theory
- Hypothesis
- Observation
- Confirmation
- Observation
- Pattern
- Hypothesis
- Theory
Subscribe to:
Posts (Atom)