Post-Graduate theory examination: Should we relook into the assessment tool?

In this editorial, I like to share some of my views and concerns regarding the MD-Community Medicine theory examination conducted by The West Bengal University of Health Sciences. Though this review focuses particularly on the Community Medicine speciality of the WBUHS, some of these may be relevant for other specialities and some other Universities as well. Opinions expressed here are based on my experiences as a paper-setter and examiner for MD examinations of the WBUHS and some other Universities for several years.

As a starting point for the discussion, I have framed the following two questions for MD- Community Medicine theory examination:

1. ‘Since the landmark work of Louis Pasteur, there has been substantial advancements in vaccine developments and technologies for vaccine quality assurance’ - Justify the statement with special reference to the immunization programme of India. [20]

2. Enlist the key interventions implemented under the National Strategic Plan for TB Elimination (2017-2025) in terms of "Detect-- Treat--Prevent--Build” components. What are the rationales for the newer inputs in this NSP compared to erstwhile RNTCP? [10+10=20]

As a paper-setter, I may be very much inclined to set these questions for the summative evaluation of MD students, as these has got immense relevance to the core competencies expected from them. The questions may test the following: (a). Knowledge of the evolution of a public health action, from where it started, and where it stands today, (b). In-depth understanding of National health policies and programmes in the current epidemiological, technological, financial and other resources’ context, (c). Skills for writing precisely without missing the key points, and (d). Power of synthesizing multiple facts, and arriving at a conclusion.(1,2)

However, considering the current model of the theory papers of the WBUHS, it is difficult to set these questions. Since 2016, the WBUHS has adopted 10 X 10 = 100 marks format for the theory papers of PG examinations. For the above two questions, optimally complete answers as expected from PG level students will be sufficiently long, and will consume substantial time of the 3-hour duration allotted for the theory papers. In my opinion, at least 20 marks may be rational for the above questions. I have no knowledge of what prompted the WBUHS to shift to this 10 X 10 marks model in 2016. I am not aware of any consultation in the Boards of Studies or any other equivalent faculty forum of the WBUHS before arriving at such a major shift in theory question pattern. Some other Universities in India moved to the 10 X 10 marks format much earlier, but this cannot be a justification for the WBUHS to follow this same. During 1980s/1990s, when we were MD (then SPM) students under the Calcutta University, we faced the 25 X 4 = 100 marks format in theory papers. That was undoubtedly against any pedagogical principle. Since the WBUHS took over the PG studies in West Bengal, format for PG theory questions was as follows: Two essay type question (2 X 20 = 40 marks) + Five short notes (5 X 6 = 30 marks) + Four brief answers (4 X 7½ = 30 marks) = 100 marks. In my opinion, this earlier model was a better assessment tool, as it allowed an optimum mix of long and short answer type questions. In the newer model followed since 2016, practically there is no scope of setting any long ‘essay type’ questions.

With the restriction of 10 marks, it is almost impossible to assess a PG student’s wholesome knowledge on any public health problems or programmes. Let me consider the question on TB above. At UG level, for an average student, some knowledge of the TB disease burden, epidemiology, outline of the TB diagnosis and case management protocols, and prevention strategies may be considered sufficient. But at PG level, a student must be conversant with the evolving epidemiological scenario over time and newer technologies, and, in response to that, shifts that are made in the guidelines, programme management strategies, surveillance systems and data management mechanisms.(3,4) PG students must have clear understanding of the rationale for the strategic changes over the years. For only 10 marks, mostly a fragment or part knowledge of any broad area of the syllabus may be assessed. Another fall-out of this is the paper-setters sometimes are not keeping in mind the restriction of 10 marks, and setting some questions for which 10 marks were unjust and 15/20 marks would have been rational.

Essay type questions are age-old tool for evaluation. There are several flaws of essay type questions, like lack of objectivity, variable reliability, difficulties of standardization for gradation, etc.(5) In spite of all these, essay type questions have several unique benefits. It has the ability to test in-depth understanding of the public health problem and interventions.(4,5) It may assess the students’ power of synthesizing and interpreting variety of information and data, capacity of examining a hypothesis with reasoning, and place arguments for justifying a conclusion.(5,6) Essay type questions are perhaps the only tool to judge one’s skills of presentation, capacity of free expression, sequencing, and arriving at a conclusion. The skill of writing optimally and precisely on a topic where large quanta of facts are available, and that too within the restricted time available during theory examination, is a great competence.(5,7) Considering all these qualities, carefully constructed long questions may maximize the effectiveness of the assessment tool. PG assessment tool must be able to test comprehensively the higher cognitive capacities, which, beside recall, will also test the interpretation and problem solving skills.(5,8)

For the above qualities, essay type questions still remain a valuable assessment tool for higher studies in many Universities in India and abroad. Any assessment tool must be appropriately aligned with the course objectives as well as the core competencies expected from the learners.(5,7,8) The competencies are contextual, and will depend on the epidemiological, demographic, socio-economic and resources scenario of the country.(1,5) The core competencies expected from our PG students include thorough academic and applied understanding of the public health problems, programme planning and management strategies that are practicable and operationally feasible in the national context. To assess such competencies, I strongly feel the present 10 X 10 = 100 model should be revised and long questions (essay type and modified essay questions) should be re-introduced for PG theory papers.

The next issue I will deal with is a review of the theory questions to find out the proportion of questions from different syllabus areas. I have compiled and analysed questions of MD- Community Medicine examinations conducted by the WBUHS in last 14 years’ (2008 to 2021). This review considered 18 MD examinations (14 regular & 4 Supple.), total 72 theory papers, and finally included 731 questions.

For compilation, the questions were classified into groups of related topics (based on my perception, somewhat empirically). However, there were certain difficulties in classification. For example, questions like "What are the policies and strategies under NIPI to control the problem of Anaemia in India?” - This may be classified as a question from ‘Nutrition’ as well as from ‘National health programme’. The ultimate objective of this analysis was to obtain a quantitative measure of the relative weightage given by the paper-setters to different syllabus areas.

Table-1: Health management information system, a major tool for public health management, should get more priority. Other areas of the syllabus were well represented. Table-2: Questions on specific disease problems and National control programmes were remarkably low. Only around 12% of questions were from this segment. Looking at such a scenario, I have done specific disease- wise compilation, instead of compiling this as a group. Very few questions were on some major public health problems like Measles, Diarrhoeal diseases, Rabies, Leprosy, Dengue, JE. There were no questions on Diphtheria, Rubella, Food poisoning, Helminthiasis, Kala-azar, Tetanus, etc. Disease epidemiology and National disease control programmes is a significant component of the expected competencies of a public health professional and must get higher weightage in the assessment process.

Table-3: Neonatal health, child health, and immunization may need more emphasis in the assessment. One high priority area, cold chain and vaccine logistics management, has been remarkably neglected. Questions on scientific writing should be more frequent. Dissemination of research findings through publications and capacity for critical appraisal of published literature are valuable competencies of the PG students.(6,8) This must be reflected in the assessment tool.

While reviewing 731 questions, I came across some flaws and discrepancies in several questions. Some of the questions, in my opinion, were qualitatively poor in terms of objectivity, framing, etc. Following are a few examples:

• Short note on ‘IYCF’ [5 marks] - IYCF guidelines (MoHFW) is a 26 page document with 10 hand-outs for training. As no specific component of IYCF, like relevance of IYCF/feeding during illness/ how to address myths & challenges etc. has been asked, it may be difficult for an examinee to plan an answer for only 5 marks. This may lead to variable interpretations by the students, as well as by the evaluators.

• Short note on ‘Qualitative research design’ [5 marks] - For a good quality answer, 5 marks is insufficient.

• ‘Detail the National Nutrition Mission’ [10 marks] - Too wide, sentence structure could have been better, scope of variable interpretations.

• ‘What is meant by Quality Assurance? Explain quality assurance measures in relation to primary health care.’ [10 marks] - PG level answer will demand more marks.

• ‘India is a hot seat for Medical Tourism- briefly validate’ - The term ‘validation’ has a specific connotation in health science. ‘Explain’ or ‘Justify’ would have been better options.

These are only few examples to disseminate the message. There were several such questions with variety of imperfections. Some of these also justify the need for reintroduction of long questions. Also, university appointed moderators should play an effective role in ensuring quality of the questions.

Another frequently observed problem was non-adherence to the paper division as recommended in the syllabus while setting the questions. This may put the examinees into difficulties. The current paper division of the WBUHS curriculum of MD-Community Medicine itself have some overlapping topics. For example, the epidemiology (study methods part), research methods and biostatistics are in three different papers. There are several inter-related areas in these three syllabus components, and the lines of demarcation are often obscure. In the ‘Competency based post graduate training programme for MD in Community Medicine’ published by MCI, all these three are included in the same paper (Paper-I). It may be better to modify the WBUHS curriculum to align it with the MCI curriculum regarding paper division.

A quality assessment tool must be able to judge the potential of a MD-Community Medicine student to function in future as an academician, a researcher, a teacher, a good communicator, a collaborator and a programme manager.(1,7,8) PG training should make them a professional with the skills and attitude to contribute effectively to the public health system. It may not be always possible to assess all the cognitive, affective and psychomotor domains perfectly within the framework of the assessment guidelines of the MCI / NMC and the Universities. But we must attempt to develop and use, as far as practicable, a comprehensive and quality assessment tool to evaluate the extent of attainment of the core competencies by our PG students.

In this editorial, I have presented my observations and concerns on certain weaknesses of the PG assessment tools related to the theoretical examination. I expect the faculty members of Community Medicine involved in PG examinations as paper-setters, moderators and examiners will consider the issues raised here. For arriving at a collective opinion, consultation in some appropriate forum like IAPSM may be a rational step forward. Finally, University authorities may be requested to consider necessary modifications in the current modalities, and to develop and follow a comprehensive assessment tool for the summative evaluation of PG students.