Validity and Reliability of a Teacher-Made Narrative Writing Assessment Instrument at SMPN 1 Sungguminasa

Introduction

Assessment is an essential part of teaching and learning because it provides information about students’ achievement and helps teachers determine whether learning objectives have been reached. In English language teaching, assessment is particularly important because language competence consists of several related abilities. Among these abilities, writing is one of the most complex because students need to develop ideas, organize information, select appropriate vocabulary, construct grammatical sentences, and apply correct writing conventions.

Writing assessment requires more than simply asking students to produce a text and assigning a score. The assessment task needs to represent the ability being measured, while the scoring criteria need to provide an appropriate basis for evaluating students’ performance. Weigle (2002) emphasizes that writing assessment involves decisions about the nature of writing ability, the design of writing tasks, and the procedures used to score written performance. Therefore, the quality of an assessment instrument is closely related to the quality of the evidence obtained from students’ writing.

Narrative writing is particularly relevant in English language learning because it requires students to communicate events in a structured and meaningful way. A narrative commonly involves a setting, characters, a sequence of events, a problem or complication, and a resolution. Students may also be expected to demonstrate characters’ responses to events and communicate a lesson or social function. Consequently, an assessment of narrative writing needs to consider both the structure of the narrative and the language used to construct it.

Teacher-made assessment instruments are widely used in classroom contexts because teachers can develop tasks according to their learning objectives, curriculum, classroom conditions, and students’ characteristics. This flexibility is valuable because classroom assessment can be adapted to the specific learning context. However, a teacher-made assessment is not automatically valid or reliable simply because it has been developed by a teacher. An assessment may appear relevant but still contain unclear criteria, incomplete coverage of the intended competence, or scoring procedures that produce inconsistent results.

Validity is therefore a central concern in assessment. Messick (1995) conceptualizes validity as an integrated evaluation of evidence and theory supporting the interpretation and use of assessment scores. From this perspective, validity is not simply a characteristic that belongs to a test. Rather, evidence must support the interpretations and uses made from the resulting scores. Content is one important source of validity evidence, particularly when an assessment is designed to represent a specific domain of knowledge or performance.

Kane (2013) further explains validity through an argument-based approach. In this approach, claims made from assessment scores need to be supported by evidence. Thus, when a teacher uses students’ writing scores to make a judgment about their narrative writing competence, the task and scoring procedure should provide sufficient evidence to support that judgment.

Reliability is another important consideration. An assessment should produce sufficiently consistent results for its intended purpose. In analytic writing assessment, consistency is particularly important because students’ writing is evaluated across several criteria. Cronbach’s Alpha is commonly used to examine internal consistency among related assessment criteria. However, internal consistency and inter-rater reliability represent different forms of evidence. Internal consistency concerns the relationship among assessment criteria, whereas inter-rater reliability concerns the consistency of scores assigned by different raters.

The present study focuses on a teacher-made narrative writing assessment instrument used at SMPN 1 Sungguminasa. The instrument was designed to assess Grade IX students’ narrative writing competence. It consisted of a narrative writing task and an analytic scoring rubric covering five major criteria: Content, Organization, Vocabulary, Grammar, and Mechanics. The writing task required students to produce a narrative based on The Multiverse Adventure of Timun Mas.

The content of the assessment was also examined through ten narrative writing components: describing the time, describing the place, introducing the main character, describing the sequence of problematic events, describing characters’ reaction or evaluation of events, providing a solution to the problem, stating character change or what the character has learned, using appropriate language components, applying mechanics, and demonstrating the social function of narrative writing.

Although teacher-made assessments are an important part of classroom assessment, empirical examination of their quality remains necessary. In particular, there is a need to determine whether the assessment content adequately represents the intended competence and whether its scoring criteria function consistently. Therefore, this study aimed to analyze the validity and reliability of the teacher-made narrative writing assessment instrument used to assess Grade IX students at SMPN 1 Sungguminasa.

The research question was:

How valid and reliable is the teacher-made narrative writing assessment instrument used to assess Grade IX students’ narrative writing competence at SMPN 1 Sungguminasa?

The study focused specifically on content validity and internal consistency reliability. This distinction is important because the study does not claim to establish every possible source of validity evidence or inter-rater reliability.

Research Method

Research Design

This study employed a quantitative descriptive research design. The design was selected because the purpose of the study was to describe and evaluate the measurement quality of an existing teacher-made assessment instrument rather than to test the effect of an instructional treatment. The study examined two main aspects of the instrument. First, content validity was investigated through expert judgment. Second, reliability was examined through internal consistency analysis of the five analytic scoring criteria. Descriptive statistics were also used to summarize students’ writing performance. The design was therefore focused on obtaining empirical evidence concerning the quality of the assessment instrument as it was used in the classroom.

Subject

The subjects of the study were 35 Grade IX students at SMPN 1 Sungguminasa. The students completed the teacher-made narrative writing assessment. Five experts were involved in the content validation process. They evaluated the relevance and adequacy of the components included in the assessment. In addition, five raters independently assessed the students’ narrative writing products using the same analytic scoring rubric. The involvement of five raters was intended to provide multiple assessments of students’ writing performance. However, as explained later, the Cronbach’s Alpha analysis in this study concerns the internal consistency of the five scoring criteria and should not be interpreted as a measure of agreement among the five raters.

Instrument

The main instrument was a teacher-made narrative writing assessment consisting of a writing task and an analytic scoring rubric. The writing task required students to produce a narrative text entitled “The Multiverse Adventure of Timun Mas.” Students were expected to organize their narratives according to the basic narrative structure of orientation, complication, and resolution. The analytic rubric consisted of five criteria: Content, Organization, Vocabulary, Grammar, and Mechanics. Each criterion contributed 20 points to the total score, resulting in a maximum score of 100.

The instrument was examined by five experts using ten content components related to narrative writing. These components represented both narrative content and language-related aspects. The ten components were:

1. describing the time;

2. describing the place;

3. introducing the main character;

4. describing the sequence of problematic events;

5. describing characters’ reaction or evaluation of the events;

6. providing a solution to the problem;

7. stating character change or what the character has learned;

8. using appropriate vocabulary and sentence structures;

9. applying appropriate mechanics; and

10. demonstrating the social function of narrative writing.

This combination allowed the assessment to consider not only language accuracy but also students’ ability to construct a meaningful narrative.

Research Procedure

The research procedure consisted of five connected stages: planning, action, observation, reflection, and final data analysis. These stages describe the practical sequence through which the assessment was prepared, implemented, examined, and evaluated. They do not represent classroom action research cycles or an intervention model. Instead, they were used to organize the assessment evaluation process systematically.

Planning

At the planning stage, the researcher identified and collected the teacher-made narrative writing assessment instrument used at SMPN 1 Sungguminasa. The researcher examined the writing task, scoring rubric, narrative components, and assessment criteria. The researcher also prepared the expert validation sheet for evaluating the relevance of the assessment components. The validation sheet included the ten narrative writing components and was designed to obtain judgments from five experts. In addition, the researcher prepared the procedures for administering the writing task, scoring the students’ writing, organizing the data, and conducting the statistical analysis. The scoring procedures were standardized so that the five raters used the same assessment criteria.

Action

The action stage involved the implementation of the teacher-made narrative writing assessment. The assessment was administered to 35 Grade IX students at SMPN 1 Sungguminasa. Students were asked to produce a narrative text based on the topic provided in the assessment. They were expected to develop the story using appropriate narrative elements, including characters, setting, problematic events, and resolution, while also demonstrating appropriate vocabulary, grammar, organization, and mechanics. After the students completed the task, their writing products were collected and prepared for assessment. The five raters then independently evaluated the students’ writing using the analytic rubric. Each writing product was assessed according to Content, Organization, Vocabulary, Grammar, and Mechanics. The resulting scores were recorded and organized for further analysis.

Observation

At the observation stage, the researcher monitored the implementation of the assessment process and documented relevant information concerning the administration of the task and scoring process. The researcher examined whether the assessment was implemented according to the intended procedure and whether the students were able to respond to the task as expected. The students’ writing products were also observed in relation to the narrative components represented in the assessment. Particular attention was given to students’ ability to develop narrative content, organize events, use vocabulary and grammar, and apply appropriate mechanics. The observation stage helped provide contextual information for interpreting the quantitative results.

Reflection

The reflection stage involved reviewing the results of the expert validation, student assessment, and scoring process. The researcher examined the strengths and areas requiring improvement in the teacher-made assessment instrument. Particular attention was given to components that received lower expert agreement and to the statistical contribution of each scoring criterion. The reflection indicated that most narrative components were strongly supported by the experts. However, the components concerning characters’ reaction or evaluation and character change or lesson learned obtained slightly lower expert agreement than the other components. These findings were considered important for future revision of the assessment descriptors. The purpose of the reflection was not to conduct a second instructional cycle but to identify areas in the assessment instrument that could be clarified or improved.

Data Analysis

The data were analyzed quantitatively using descriptive and inferential measurement statistics appropriate to the purpose of the study.

Content Validity

Content validity was analyzed using the Item-Level Content Validity Index (I-CVI) and Scale-Level Content Validity Index based on the average method (S-CVI/Ave). The I-CVI was calculated by dividing the number of experts who considered an assessment component relevant by the total number of experts. The S-CVI/Ave was calculated by averaging the I-CVI values across all ten components. This approach was used to quantify the level of agreement among the five experts regarding the relevance of the assessment content.

1. Reliability

Reliability was examined using Cronbach’s Alpha. The analysis focused on the internal consistency of the five scoring criteria: Content, Organization, Vocabulary, Grammar, and Mechanics. Corrected item-total correlation was examined to determine the relationship between each criterion and the overall score. Alpha if Item Deleted was also examined to determine whether removing a criterion would increase or decrease the overall internal consistency.

1. Descriptive Statistics

Descriptive statistics were used to summarize students’ writing performance. The analysis included the mean, frequency, percentage, and standard deviation of the scores. The students’ performance was also classified according to the assessment categories used in the study. These statistics were used to provide a general description of the students’ narrative writing performance and to support interpretation of the assessment results.

Research Findings And Discussion

Content Validity

The content validity analysis showed strong agreement among the five experts regarding the relevance of the assessment components.

Eight of the ten components obtained an I-CVI of 1.00, indicating complete agreement among the five experts. Two components obtained an I-CVI of 0.80, namely characters’ reaction/evaluation and character change/lesson learned. The overall S-CVI/Ave was 0.96, demonstrating very high content validity. This finding indicates that the content of the teacher-made assessment was strongly aligned with the intended domain of narrative writing. The assessment did not focus exclusively on grammatical accuracy but represented several important aspects of narrative competence, including story development, character development, language, mechanics, and social function. The result also supports the view that assessment content should provide an adequate representation of the domain that is intended to be measured. Within Messick’s validity framework, evidence concerning the content and structure of an assessment contributes to the broader argument for interpreting its scores.

The S-CVI/Ave of 0.96 provides strong evidence that the assessment content was considered relevant and representative by the experts. However, content validity should be understood as one source of validity evidence rather than evidence that establishes every aspect of validity.ReliabilityThe reliability analysis demonstrated very high internal consistency among the five scoring criteria.

The Cronbach’s Alpha coefficient was 0.930, while the standardized Alpha was 0.938. These coefficients indicate that the five scoring criteria functioned consistently as components of the overall assessment. The result suggests that Content, Organization, Vocabulary, Grammar, and Mechanics were sufficiently related to students’ overall writing performance. However, the high coefficient should not be interpreted to mean that the five criteria measure exactly the same feature. Each criterion represents a distinct dimension of writing competence. The coefficient instead indicates that the criteria operate coherently within the overall scoring system.

Corrected Item-Total Correlation

The corrected item-total correlations ranged from 0.774 to 0.889. Vocabulary obtained the highest correlation at 0.889, while Content obtained the lowest at 0.774. Although the coefficients differed, all five criteria demonstrated strong relationships with the overall assessment score. The findings suggest that each criterion made a meaningful contribution to the measurement of students’ narrative writing performance. No criterion appeared to function independently from the overall writing construct.Alpha if Item Deleted

All Alpha if Item Deleted values were lower than the overall Alpha of 0.930. This finding indicates that removing any of the five criteria would reduce the internal consistency of the assessment. Therefore, all five criteria should be retained. The results also show that the five criteria complement each other. Content, Organization, Vocabulary, Grammar, and Mechanics provide different types of information about students’ writing, while together they form a coherent analytic scoring system.

Students’ Narrative Writing Performance

The assessment was administered to 35 Grade IX students. The overall classification of student performance showed that most students were located in the middle performance categories.

The largest group consisted of students in the Fair category, with 20 students or 57.14%. Ten students, or 28.57%, were classified as Poor, while three students, or 8.57%, were classified as Good. Two students, or 5.71%, were classified as Very Poor, and no student was classified as Very Good. The distribution indicates that most students were able to produce narrative writing at a basic to moderate level but still required improvement to reach a higher level of writing performance. This result is relevant to the assessment quality because the instrument was able to differentiate students’ writing performance across several levels. The use of an analytic rubric also allowed performance to be considered across multiple writing dimensions rather than through a single overall impression.

Discussion

The main finding of this study is that the teacher-made narrative writing assessment instrument demonstrated very high content validity and very high internal consistency.

The S-CVI/Ave of 0.96 indicates that the experts strongly agreed that the assessment content was relevant. This result is important because an assessment should represent the knowledge and skills that it intends to measure. The ten content components used in the validation represented both narrative-specific features and general writing requirements.

The inclusion of time, setting, characters, problematic events, resolution, and character development allowed the assessment to capture important features of narrative writing. At the same time, the inclusion of language components and mechanics ensured that students’ ability to communicate the narrative through English was also considered.

This finding is consistent with the broader theoretical view of validity proposed by Messick, in which content representation forms part of the evidence used to support the interpretation of performance assessment results.

The finding is also consistent with Kane’s argument-based approach because the interpretation of a student’s writing score requires evidence that the task actually elicits the performance intended to be evaluated. In the present study, the strong expert agreement provides initial evidence supporting the claim that the assessment content is relevant to narrative writing competence. However, as Kane emphasizes, more extensive claims require more extensive evidence.

The two components that received an I-CVI of 0.80 deserve particular attention. These were characters’ reaction/evaluation and character change/lesson learned. The lower agreement does not mean that these components are inappropriate. Rather, it may indicate that they are more difficult to define or interpret consistently than basic narrative elements such as setting or characters.

For example, a character’s reaction may be expressed through dialogue, action, thought, or direct description. Similarly, character change may be explicitly stated or indirectly demonstrated through changes in behavior. Therefore, these components may benefit from clearer descriptors and examples.

The reliability results provide another important finding. The Cronbach’s Alpha of 0.930 indicates very high internal consistency. The standardized Alpha of 0.938 provides similar evidence. 

The result supports the use of the five-criterion analytic rubric. In writing assessment, analytic scoring is useful because it allows different aspects of writing performance to be evaluated separately. 

Weigle’s work on writing assessment emphasizes the importance of appropriate scoring procedures when writing performance is evaluated. 

The corrected item-total correlations provide further support. The coefficients ranged from 0.774 to 0.889, demonstrating that all five criteria were strongly associated with the overall assessment score. Vocabulary had the highest coefficient, 0.889. This suggests that vocabulary performance was strongly related to overall writing performance in this sample. However, this should not be interpreted as evidence that vocabulary is theoretically more important than Content, Organization, Grammar, or Mechanics. The coefficient only describes the statistical relationship observed in this particular assessment.

Content had the lowest coefficient, 0.774, but this was still a strong relationship. Therefore, Content remained an important component of the assessment and should not be removed.

The Alpha if Item Deleted results reinforce this conclusion. All deletion coefficients were lower than the overall Alpha. If Content were removed, for example, Alpha would decrease to 0.924. If Vocabulary were removed, Alpha would decrease further to 0.900. This indicates that every criterion contributes to the overall consistency of the assessment.

The results therefore support retaining the complete five-criterion rubric.

An important methodological issue concerns the five raters involved in the study. Because five raters assessed the students’ writing, it may be tempting to interpret the Cronbach’s Alpha of 0.930 as evidence of inter-rater reliability. Such an interpretation would be incorrect.

Cronbach’s Alpha in this study was calculated across the five assessment criteria, not across the five raters. Therefore, the coefficient indicates internal consistency among Content, Organization, Vocabulary, Grammar, and Mechanics.

Inter-rater reliability requires a separate analysis based on the scores assigned by the different raters. An Intra class Correlation Coefficient would be more appropriate for examining the degree of consistency or agreement among the five raters.

This distinction strengthens the methodological interpretation of the findings. Rather than claiming that the instrument has demonstrated all forms of reliability, the study can accurately state that it has demonstrated very high internal consistency. 

The findings also have practical implications for English teachers. First, teacher-made assessments can provide useful classroom evidence when their content and scoring procedures are systematically evaluated. Second, teachers should ensure that narrative writing tasks represent the essential features of the genre. Third, analytic rubrics should provide clear and observable criteria. Fourth, assessment instruments should be periodically reviewed rather than being treated as permanently fixed.

The findings also demonstrate the value of expert validation before an assessment is used for important classroom decisions. Expert judgment can identify components that may be theoretically relevant but insufficiently clear for practical scoring.

The study further shows that validity and reliability should be treated as complementary. The S-CVI/Ave of 0.96 indicates strong expert agreement about content, while the Cronbach’s Alpha of 0.930 indicates strong internal consistency. Neither statistic alone is sufficient to establish complete assessment quality.

This interpretation is consistent with modern validity theory. Validity concerns the evidence supporting interpretations and uses of scores, and different claims require different types of evidence.

Therefore, the present findings support a limited but meaningful validity argument: the assessment content is strongly supported by expert judgment, and its five scoring criteria demonstrate high internal consistency. Further evidence would be needed to make broader claims concerning construct validity, consequential validity, generalizability, and inter-rater reliability.

Conclusion

This study examined the validity and reliability of a teacher-made narrative writing assessment instrument used to assess Grade IX students at SMPN 1 Sungguminasa.

The content validity analysis produced an S-CVI/Ave of 0.96, indicating very high expert agreement concerning the relevance of the assessment components. Eight of the ten components obtained an I-CVI of 1.00, while two components—characters’ reaction/evaluation and character change/lesson learned—obtained an I-CVI of 0.80. These two components should therefore be refined through clearer and more specific descriptors.

The reliability analysis produced a Cronbach’s Alpha of 0.930 and a standardized Alpha of 0.938, indicating very high internal consistency among the five scoring criteria. Corrected item-total correlations ranged from 0.774 to 0.889, and all Alpha if Item Deleted values were lower than the overall Alpha. Therefore, all five criteria—Content, Organization, Vocabulary, Grammar, and Mechanics—contributed positively to the consistency of the assessment.

Overall, the teacher-made narrative writing assessment instrument demonstrated strong evidence of content validity and very high internal consistency. It can therefore be considered appropriate for assessing Grade IX students’ narrative writing in the investigated classroom context.

However, the findings should not be interpreted as evidence of complete validity or inter-rater reliability. Future research should examine additional sources of validity evidence and conduct a separate inter-rater reliability analysis, particularly because five raters were involved in scoring the students’ writing.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.Informed Consent Statement

Participation in this study was voluntary. Before the research was conducted, the participants were informed about the purpose and procedures of the study, particularly the administration of the teacher-made narrative writing assessment. The students were informed that their participation involved completing a narrative writing task and that their writing products would be used only for research purposes related to the analysis of the validity and reliability of the assessment instrument. The participants’ identities and personal information were kept confidential and were not disclosed in the reporting of the research findings. Participation or non-participation did not affect the students’ academic standing or classroom assessment. The research was conducted with the necessary permission from the school and relevant parties. The collected data were securely handled and used solely for academic and research purposes.

Data Availability Statement

The data supporting the findings of this study are available from the corresponding author upon reasonable request, subject to applicable ethical and institutional considerations.

Acknowledgement

The researcher would like to express sincere gratitude to the academic supervisors, expert validators, English teachers, raters, and Grade IX students of SMPN 1 Sungguminasa for their valuable support and participation throughout the research process. The researcher also thanks all parties who contributed to the completion of this study.