Background: Despite its importance for diagnosing idiopathic inflammatory myopathy (IIM) subgroups, muscle biopsy classification criteria have not been standardized nor validated. Objective: We sought to determine inter-rater variability in diagnosing individual muscle biopsy abnormalities. Methods: We developed a scoring tool to analyze consensus in muscle biopsy reading of an ad hoc workgroup of international experts. Twenty-four samples from suspected IIM patients were randomly selected, providing sections that were stained with standard histological and immuno-histochemical methods. Sections were made available on an online platform and experts were queried about myopathological features within four pathological domains: muscle fibers, inflammation, connective tissue and vasculature. A short clinical presentation of cases was included, and experts were asked to give a tentative diagnosis of polymyositis, dermatomyositis, inclusion-body myositis, anti-synthetase syndrome-related myositis, immune-mediated necrotizing myopathy, non-specific myositis or other disease. Fleiss’ Kappa values, scoring inter-rater variability, showed the highest agreement within the muscle fiber and connective tissue domains. Results: Despite overall low Kappa values, moderate agreement was achieved for tentative diagnosis, supporting the idea of holistic muscle biopsy interpretation rather than by adding up of individual features. Conclusion: We conclude that assessment of individual pathological features needs to be standardized and harmonized, and should be measured for sensitivity and specificity for subgroup classification. Standardizing the process of diagnostic muscle biopsy reading would allow identification of more homogeneous patient cohorts for upcoming treatment trials.