Developing and evaluating a professional development plan pilot for doctors in unaccredited posts: A pilot study

Jill Thistlethwaite ^{1, 2}, Lara McGirr¹, Toni Vial¹

Abstract

Purpose

In Australia, doctors from their third-year post graduation who are not on specialist training pathways frequently work in unaccredited posts with varying amounts of education and support. In 2019, the New South Wales Ministry of Health (NSW Health) and the Health Education and Training Institute (HETI) agreed on a pilot process for these doctors to develop a professional development plan (PDP). The pilot aimed to implement the process and evaluate its feasibility and acceptability.

Methodology/approach

The process was piloted at four sites in 2021. The evaluation methodology was informed by the non-adoption, abandonment and challenges to scale-up, spread and sustainability (NASSS) framework with data derived from site meetings, interviews with doctors in unaccredited positions and PDP supervisors, and analysis of PDPs and time required.

Findings

A total of 42 doctors undertook the PDP process, of whom 25 were interviewed. Of the 28 supervisors recruited, 13 were interviewed. Three sites reported successful implementation, with most doctors having a PDP in progress. Despite challenges associated with the diversity of the workforce and workplaces, all sites were supportive of the process being rolled out with appropriate resourcing.

Research implications

The research findings indicated that embedding a PDP process more widely across the state will be complex due to the diversity of the workforce and clinical workplaces.

Practical implications

The PDP process, while acceptable and feasible, needs to adapt to local circumstances, including the workforce, supervisory capacity and experience, individual doctor needs and available resources.

Originality

The evaluation supports the need for a supported PDP process for doctors in unaccredited positions.

Limitations

The findings may not be transferable to all NSW Health facilities or to other states or territories. Doctors who consented to be interviewed were more likely to be positive about the process than those who did not. The study did not include a cost evaluation or explore cost-effectiveness due to the short time frame.

Keywords: professional development plans; continuing professional development; junior doctors; service registrars; education evaluation

Introduction

In New South Wales (NSW), Australia, doctors in their first two years after graduation (postgraduate year [PGY] 1 and 2) are referred to as prevocational and undertake an educational program with supervisory support. Following this, doctors are typically either accepted into a vocational specialist structured training program in accredited hospital posts or take up an unaccredited position. In NSW, the former are referred to as registrars, while the latter are known variously as career medical officers, senior resident medical officers or unaccredited trainees. The National Medical Workforce Strategy 2021-2031 refers to doctors in unaccredited positions as service registrars (Australian Government Department of Health 2021). The recent rise in the number of medical students in Australia without a concomitant increase in training posts has led to a rise in the number of service registrars. These include those who plan to apply for a specialist training position, have left a training program, are uncertain of their preferred career or have decided not to enter specialist training. Service registrars may also be international medical graduates.

Service registrars have historically worked long hours in public hospitals, carrying out similar tasks to specialist registrars but without the same levels of supervision, education and limits on overtime. The importance of their contribution to healthcare is not always given the recognition and respect it deserves by medical and other health professionals, and service registrars are prone to exploitation (Australian Government Department of Health 2021). Because of concerns about these doctors, in 2008, the NSW Ministry of Health (NSW Health) established the Hospital Skills Program (HSP) through its education organisation, the Health Education and Training Institute (HETI). It was a flexible training and development program designed to provide a structured set of learning outcomes in a set of HSP modules, launched between 2010 and 2012. They were based on authentic clinical contexts and included emergency medicine, aged care, mental health, children's health, hospital medicine, women's health, addiction medicine, rural medicine, sexual health and Aboriginal health. However, uptake has been poor, partly because of its detail and lack of strategic direction for its use. Moreover, the modules have not been updated in line with clinical guidelines or practice over the last nine years. The HSP was also intended to include a professional development process undertaken by the local health districts (LHDs) in which the doctors were employed. This process consisted of each HSP doctor formulating a learning plan, defined as a set of learning outcomes based on their role and educational needs, planned activities to achieve the outcomes and evidence of learning. Evidence of learning incorporated work-based assessment (WBA), comprising observation and evaluation of the doctors' performance during clinical activities.

HETI was aware of considerable concerns about the practicalities of implementing such a process across the state, particularly in accessing appropriate training, supervision and review for doctors working without consistent on-site supervision. In 2013, a pilot professional development plan (PDP) process was undertaken at one hospital, which found that while there was variability in the engagement of the doctors, the process was considered valuable and could be modified for other areas of practice. An essential factor for success was a positive supervisory relationship in which supervisors acknowledged the educational and other needs of the non-specialist doctors. However, the time and resources required for arranging and completing WBA limited feasibility, and there were general misconceptions about WBA. The pilot report recommended trialling the process at more sites before a statewide rollout (Ozolins et al. 2014).

The HSP continued to provide funds for education for all doctors in non-accredited posts in NSW based on their perceived needs. However, this workforce is diverse, with unaccredited posts across multiple specialties and locations. Education has not been tailored to individual needs and career progression. The Australian media has published stories about these doctors' heavy workload (Lindsay 2019; McKinnell 2019). In response, the Australian Medical Association (AMA) of NSW, an independent body representing the state's medical profession, reported that its council was 'concerned about the growing cohort of junior doctors working in unaccredited registrar positions and believes that this is a waste of human capital and may be affecting the provision of safe, high-quality patient care' (AMA NSW 2019).

In 2018, HETI agreed that more support for PGY3 to PGY5 doctors in non-accredited posts was required. These doctors would now be referred to as hospital non-specialist program (HNSP) doctors. In 2019, an agreement was reached to pilot a supported PDP process in the state. In addition, the MBA announced that all registered medical practitioners, with a few exemptions, must have a written and ongoing PDP from 1 January 2023 (Medical Board of Australia 2021).

By definition, a PDP ('trainee' or 'learner' may be substituted for 'employee'):

gives an overview of the competencies the employee worked on in the past and which competencies the employee is planning to work on in the future … is composed by the employee himself … and is used as the basis for or to structure the conversations with the supervisor or the coach, who provides the employee with feedback and stimulates the employee's reflection. (Beausaert et al. 2011, p. 236)

Before piloting, the process was widely discussed and developed with stakeholders across NSW Health. The consensus was that the NSW PDP process aims to support HNSP doctors in a continuous cycle of improvement and learning by determining their current level of competence and performance and facilitating the identification and achievement of learning outcomes related to their specific needs and career plans. We identified conversations with supervisors as an important part of the process. They offered an opportunity for the HNSP doctors to reflect on the learning needs for their current clinical roles, future roles and career aspirations with their supervisors' guidance. The example template for the pilot included the typical components of PDPs (Challis 2000), but these were not mandatory. It was recommended that learning outcomes be written as specific, measurable, achievable, realistic and timely (SMART) goals (Doran 1981). The supervisor's role included meeting with the HNSP doctor, supporting the identification of learning goals and relevant learning activities to meet the goals, discussing evidence of learning and advising on career progression.

PURPOSE

The aims of the PDP pilot over 13 weeks were to develop, implement and evaluate the process to enhance and sustain the training and satisfaction of the HNSP workforce in NSW. The evaluation focused on the process's feasibility and acceptability rather than longer-term outcomes such as learning achievement, career development and patient outcomes.

METHODOLOGY AND APPROACH

The development and delivery of the HNSP require a change in management approach. Greenhalgh and Papoutsi (2019) have described the dissemination of innovation across a healthcare system as challenging but achievable by considering three different logics of change and their underlying theories: mechanistic (implementation science), ecological (complexity science) and social (social science).

Implementation science focuses on the uptake of evidence-based practices into routine practice (Nilsen 2015). In the case of the HNSP, evidence-based education must inform the program. The intervention (the PDP process) must be clearly defined and implemented, considering how individual and organisational behaviour may be changed. Ideally, a small-scale trial in a few settings is evaluated (i.e., a pilot process).

Complexity science studies complex systems that are uncertain, unpredictable and emergent (Greenhalgh & Papoutsi 2019). Healthcare settings are complex in nature and dependent on diverse groups of health professionals, educators and administrators. To succeed in implementing the HNSP, the unpredictability of the workplace and the need to modify the program for local contexts must be recognised.

Social science aims to explore what people believe, why they work the way they do, how they interpret others' actions and what they draw on to achieve their (or a program's) goals. Staff work differently in different contexts; they work around problems in diverse ways and adapt innovations to their needs and resources. Thus, attempting to standardise a program as a rigid 'one size fits all' model is unlikely to be feasible or acceptable.

These three approaches informed the methods of implementation and data collection to capture the contextual factors underlying the adoption of the HNSP pilot PDP process. The framework for the evaluation was the non-adoption, abandonment and challenges to scale-up, spread and sustainability (NASSS) framework of Greenhalgh et al. (2017) (Figure 1). The NASSS framework was developed as an evidence-based, theory-informed and pragmatic framework to help predict and evaluate the success of implementing a new technology-supported health or social care program. We considered it appropriate for predicting and evaluating the implementation of other programs - in this case, the PDP process for the HNSP. The adoption and diffusion of innovations are not solely affected by individual factors such as finance, technology, staff and learners considered separately but by the dynamic interactions between them. It is these interactions that were explored in the pilot. The seven domains of the NASSS informed the evaluation of the pilot to help predict the feasibility of the process and provide recommendations to enhance the likelihood of adoption, given the current climate of doctor shortages and increased workload pressures due to the ongoing COVID-19 pandemic. The pilot evaluation questions are listed in Table 1.

Figure 1: The NASSS framework

Source: Greenhalgh T, Wherton J, Papoutsi C et al, 2017, 'Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread and sustainability of health and care technologies', J Med Internet Res, vol. 19(11): e367. doi: 10.2196/jmir.8775.

Published (and can be reproduced) under the terms of Creative Commons Attribution 4.0 licence.

Table 1: Evaluation questions informed by the NASSS domains

	Domain	Questions
1	Condition (Lack of structure and support for the hospital non-specialist medical workforce)	How is this condition perceived by participants? What are the socio-cultural factors to consider in this context?
2	Innovation (intervention) (The professional development process)	What are the important features of the process? What is required in terms of technology? What knowledge and skills are needed to take part?
3	Value proposition	What is the business case/rationale for implementation? What are the desirable outcomes, the effectiveness and cost-effectiveness of the process?
4	The adopter system	What changes are implied for staff (HNSP doctors, supervisors, support staff)? What is expected of these individuals?
5	The healthcare organisation	How does the structure of the organisation (NSW Health) in each local context affect the capacity to implement? How ready is it for change? What are the implications for the wider health team? What work is needed to implement?
6	The wider system	What is the political and policy context underpinning the implementation? What are the regulatory or legal hurdles? What inter-organisational work is required?
7	Embedding and adaptation	Is this process likely to be sustainable? What affects the likelihood of sustainability? What differences might be expected in different locations?

During the planning and implementation phases of the pilot, we used appropriate evaluation methods to determine whether the domains were simple, complicated or complex, as defined in the NASSS framework. A domain is simple if it is straightforward and predictable, complicated if it has multiple interacting components or issues and complex if it is dynamic and unpredictable: 'The more complex an innovation or the setting in which it is introduced, the less likely it is to be successfully adopted, scaled up, spread, and sustained' (Greenhalgh et al. 2017).

ETHICAL APPROVAL

Hunter New England Research Ethics Committee provided ethical approval for this project (2019/PID15113). Site-specific permission was also given.

SETTINGS AND PARTICIPANTS

Information on the pilot was shared with LHD chief executives. Expressions of interest were invited towards the end of 2020, and the panel selected four sites including metropolitan and rural areas. Funding was provided for a clinical lead and an administrator at each site, who were responsible for selecting and inviting HNSP doctors and supervisors to take part. We provided resources including guides for HNSP doctors and supervisors and the example PDP template (Figure 2), although we advised that other PDP formats could be used.

Figure 2: The suggested PDP template

	Examples
Date
Learning outcome(s) SMART (specific, measurable, achievable, realistic and timely)
Rationale: why do I need to achieve this outcome? How do I know I need to achieve this outcome? How will this improve my performance?	New post Patient's unmet needs Significant event audit Supervisor advice Review and planning meeting Interest Relevant college curriculum Career intentions
How will I achieve this outcome? How will I learn? Activities	Online learning Reading Workshop Departmental education Observation and feedback Practice Paid course
Date achieved (partial/full)
What have I learned?
How do I know I have learned? Evidence of learning	Formal certificate/assessment Observation in practice such as mini-clinical evaluation exercise, direct observation of practical skills, multi-source feedback, self-assessment supervisor feedback
Current post Application to current post

Data collection

The pilot process began in January 2021 and lasted 13 weeks (one term). A wide range of data were collected during the pilot and in the following weeks from:

• transcripts of Zoom meetings with pilot sites and the steering committee
• copies of PDPs
• supervisor and administration staff time logs
• transcripts of one-to-one Zoom interviews with supervisors and HNSP doctors
• end-of-pilot reports from each site.

Interviews were semi-structured with areas for discussion based on the NASSS framework domains (Figure 1). They were iterative in that additional topics were informed by analysis of previous interviews. Questions for further exploration also arose during the interviews themselves. Interviews lasted from 20 to 45 minutes and were conducted by the lead author.

Data analysis

Interview transcripts were analysed, synthesised and organised under the seven domains of the NASSS framework. Content analysis (Liamputtong 2020) was suitable for this study as it was not designed to develop new knowledge but to evaluate the feasibility and acceptability of the PDP process, describe the implications of the findings and develop recommendations for NSW Health. Thus, there were specific questions the text needed to answer. The study looked for patterns in the data rather than quantifying responses. Data were analysed in Taguette, a basic qualitative data software package. The content of the PDPs was quantified and described according to the suggested areas to be included in the plans. Quantitative data were extracted from time logs and reports and tabulated.

Findings

Of the 42 HNSP doctors who undertook the PDP process, 25 were interviewed. Two additional HNSP doctors at Site C who were not in the pilot were interviewed for comparison. Two at Site D who were not allocated a supervisor to start the process in time were also interviewed. Of the 28 supervisors who participated, 13 were interviewed (Table 2). This convenience sample was drawn from participants who responded to our invitation, signed a consent form and were available to be interviewed during the evaluation period. The study analysed 31 PDPs and all time logs from each site. Three of the pilot sites reported successful implementation of the PDP process with good engagement by HNSP doctors and supervisors. The team at Site D, two hospitals in a rural and remote area, was delayed in recruiting staff to the pilot director and administrator positions. This led to late dissemination of information about the pilot to hospital staff, including HNSP doctors, and no orientation event at the start of the term, resulting in minimal uptake of the process. The PDPs contained from two to 30 learning outcomes; eight had SMART goals.

Table 2: Four pilot sites and participant data

Site	A	B	C	D
Description	New critical care rotation with 15 HNSP positions in one district metropolitan hospital	HNSP doctors in different departments in one major metropolitan teaching hospital	LHD with four hospitals involved, HNSP doctors in intensive care unit and emergency departments	Rural and remote LHD with two hospitals involved and multiple departments
Number of HNSP pilot doctors	14	18	19	20**
Interviewed	8	7	9 + 2*	3
Number of pilot supervisors	10	14	4
Interviewed	3	3	3	4

*At Site C, nine HNSP doctors in the pilot were interviewed and two HNSP doctors not in the pilot were interviewed to compare.

**At Site D, there were 20 eligible HNSP doctors, only one of whom started a PDP with supervisor support. Three HNSP doctors and four potential supervisors were interviewed.

Participants' quotations are identified by the site letter (A, B, C or D), followed by 'S' for supervisors or a PGY number for HNSP doctors. For example, BS006 indicates a supervisor at Site B, and B3009 indicates a PGY3 HNSP doctor at Site B.

DOMAIN 1: THE CONDITION

The nature of the condition is a lack of structure and support for service registrars, compared to doctors on specialist training programs.

So, their progress is not really tracked or measured, and they often don't get much feedback in terms of where they are compared to their colleagues and what's expected of them and what they can do to improve. (BS004)

I don't think I expected [in PGY3] to feel quite so unguided as to what to study or what to work on. (A4005)

The diversity of the individuals within this group is high, adding to the complexity of the condition. The pilot cohort included graduates from nine Australian medical schools and international medical graduates from countries including the United Kingdom, China, Pakistan, India, Nigeria and Brazil. The HNSP doctors reported varying levels of support depending on their department, the number of trainees, the availability of supervisors or mentors and their working rotas.

DOMAIN 2: INNOVATION AND TECHNOLOGY

While the intervention (the PDP process and documentation) is not innovative in its conception or adoption in many settings, including healthcare, it is novel for this cohort. Many interviewed reported having no experience setting their own learning goals or developing individualised learning plans or PDPs. PGY4 doctors and above were more likely to report self-directed learning experience, either individually or facilitated.

Not like this where you meet with someone more experienced to help and really organises [sic] a plan. Of course, we all make plans in our minds, but that's different. So, I haven't had one before. No. (C3019)

I think I've done them before but not for a whole year, it's more just like as a junior doctor, you often work in terms of 10 weeks. Quite a few times at the start of 10 weeks, if your consultant has time, they'll sit down and be like, what do you want to get out of this term? So yeah, I think I have done them before but they're quite informal. (D3001)

Interview data strongly indicate that a timely orientation package is required to brief and support HNSP doctors before starting a PDP. Supervisors offered differing levels of support depending on each doctor's engagement with the process and ability to set, monitor and provide evidence of learning.

They were, I mean even when we introduced the PDPs at our orientation day, they said well who's going to give that to us … [we said] no, this is your chance to develop it yourself. (AS015)

Site A recommended that HNSP doctors use the My Osler application for PDP development (https://www.oslertechnology.com) and site B used Microsoft Teams rather than the paper-based template. While My Osler was well liked, particularly due to its portability, Microsoft Teams was found clunky and hard to master. Interviewees identified what a suitable platform needed.

Easily accessible by everybody. Functional. Not so many buzzwords, just really down-to-earth, simple type of stuff where you can document your roles, your plans and your outcomes and whether you achieve it. I think that's all you need. (BS006)

DOMAIN 3: THE VALUE PROPOSITION

The aim of supporting HNSP doctors is to improve patient outcomes and safety by realising the full potential of each doctor in the NSW Health system. It will be difficult to show a direct link between the introduction of the PDP process and patient care without an in-depth long-term evaluation. Thus, the return on investment is difficult to measure. However, support for doctors is likely to improve performance, and the doctors in the pilot had positive feedback about the process and its necessity. The value will need to be clear to all, including funders and all staff, prior to statewide adoption, which highlights the importance of advertising the benefits of the process. These were identified as support, structure, development guidance, career advice, mentoring and enhanced wellbeing.

I think a big positive is actually seeing - keeping a record of it, actually seeing the work you're putting [in] and what you're achieving. Because I think I've always done things before but never really recorded it and so never really felt like I was accomplishing anything. So, I think it's positive reinforcement when you see what you're accomplishing, you feel more motivated to continue. (A4005)

They get something out of it from a career perspective, but I think they also get something out of it from a wellbeing perspective. (CS001)

I do like that there's a bit more structure and the fact that the whole department's aware that I'm trying to progress with something and I'm not just trying to cruise through the year, and that's been really helpful. (B3005)

For some, the PDP process was seen as of essential value in principle, but problematic in practice, due to local difficulties with staffing levels and the senior workforce having to take on additional supervisory tasks.

DOMAIN 4: THE ADOPTER SYSTEM

Most HNSP doctors stated that they would continue the process if support was provided. However, expectations need to be realistic, and there could be a danger of losing the educational value.

Yes. I think so. Absolutely. Especially because I feel that the supervisors, they are willing to help any time (C3019)

I know I have a cynical view of top-down implemented programs because they tend to turn into tick box exercises, or they add workload onto somebody that doesn't really want the workload and you don't really get the engagement in that type of implementation. (D6014)

The PDPs analysis indicated that HNSP doctors need support to recognise learning goals and provide evidence of learning. Few PDPs included SMART learning goals. Instead, goals were broad and vague, without a rationale or timeline for achievement. Most goals were clinically focused, and few were professional or career oriented. In addition to developing new skills in self-directed learning, adopters will need to engage with feedback. Differences were noted in the HNSP doctors' comfort levels in eliciting feedback. This facet of learning needs addressing to help HNSP doctors set learning goals and gain evidence of meeting them.

It would be very rare that I'd ask someone directly for feedback. (C3010)

I think I definitely go for feedback. Especially when something has gone wrong, or I don't think I've done such a good job or I'm unsure about things. (B4001)

I've had some feedback from my supervisors and I thought that was really helpful. So, I've been asking people to tell me if I'm doing something wrong or if I can do something better and what they think is going to help me for future learning. (C3019)

It's really variable. Sometimes really well because their personal values or approach encourages the seeking of feedback for them as an individual to almost never and in the group of those who almost never seek it, I'm never sure if it's just because they don't know that they can or if they're afraid of what they may hear as well. (DS001)

The supervisors were positive about the process, but they recognised that even in a pilot with supposedly engaged trainees, the response may be variable. Some trainees would require extra input and workload. Supervisors themselves could vary in enthusiasm, and sufficient time for training for new supervisors would be required.

As a supervisor or as somebody who's trying to mentor these younger doctors, you've got to want to do it, you've got to have some interest, and you've got to show some enthusiasm. (AS001)

Some supervisors may feel that they don't have the skills to sit down and talk to trainees about professional development. So, they might feel that they need a bit of guidance in what it is they need to do. (AS005)

DOMAIN 5: THE ORGANISATION

The overall organisation is NSW Health, with multiple health districts and hospitals. In the pilot, the organisation included the local hospitals at the four sites. Within the organisation are clinical workplaces of diverse sizes, geography and capacity to innovate. Taking all sites into account, the organisation is complex. Participants at all pilot sites were highly supportive of the PDP process, including Site D, even though the process there did not run as envisaged due to communication issues. The PDP process will not deliver short-term cost savings and requires resources to introduce and support it. Local organisations will need to factor in time for trainees and supervisors in a system where non-specialist trainees are frequently used to fill gaps in work rotas.

In our team, the unaccrediteds typically do do more nights. (B4001)

At the time of being interviewed, most HNSP doctors at three sites had had either one or two meetings with their supervisor. Some had a third meeting planned. Sessions lasted from 15 minutes to one hour, with the first meeting typically longer than the second. For supervisors, time was not only spent in discussion but also on documentation and, outside meetings, organising times to meet.

It's pretty hard to find the time. (B4001)

I've negotiated some non-clinical time to be able to do that. (AS001)

Time logs indicated that the PDP process would require a minimum of 1.5 to three hours' clinical release time for a supervisor and each HNSP doctor over a 13-week term, with variable additional time for documentation and arranging meetings. Once embedded, administration should take less time than the pilot process. Depending on the number of doctors per site, it could require the equivalent of at least 0.125 full-time work (just over half a day per week).

DOMAIN 6: THE WIDER SYSTEM AND CONTEXT

Most interviewees agreed that the PDP process should become mandatory.

I think it would have to be a top-down approach for all unaccredited trainees in New South Wales, and probably for Australia, if I'm honest; but to start with, New South Wales first. I don't think it can be hospital specific. I think if you're going to have something like this, it should be done for all trainees. (AS001)

I think it's still quite flexible and you can make it quite personal so no [misgivings about being mandatory]. (D3001)

Some interviewees noted that not all departments and hospitals have adequate staffing, particularly in rural areas. They are likely to struggle with additional supervisory commitments, particularly as many potential PDP supervisors are also supervising doctors on specialist training pathways and have other teaching responsibilities.

Now you've got often non-accredited doctors who are actually most in need of support and training working in areas that are least equipped to provide the support and training. (DS003)

Both accredited and unaccredited trainees and even I have medical students from [XX] Uni who come through. I supervise them as well. (BS001)

DOMAIN 7: EMBEDDING AND ADAPTATION OVER TIME

A new process will need to adapt to ensure its sustainability and to become embedded in clinical practice. Building PDP requirements into contracts for HNSP doctors and supervisors was seen as important for success.

For it to work, I feel like having a dedicated supervisor who has this supervision part as a part of their non-clinical portfolio would be a good thing. (CS001)

The pilot lasted one term, whereas the process, if embedded, would extend at least three years (from PGY3 to PGY5) for the HNSP doctors. Ongoing supervision will need to be considered as doctors move within the organisation. It will be difficult for doctors to retain the same PDP supervisor after changing location or department. Several doctors advocated having the same supervisor for one year, while others saw the benefits of change.

Keeping the same person is definitely better, to have that continuity, I think, is really important. To move supervisors, which is something I did in [XX], was frustrating, and always felt like you never really got to know someone. (B4007)

Maybe it's a good idea to see someone else because they might have different experiences and they might have different options. They might know different courses, or they might know different people to help me organise a meeting with someone from my specialty. It might be a good idea to talk to someone with different experiences. I don't see that as a problem. (C3019)

Discussion

The ongoing tension between the education of recently qualified doctors and clinical service delivery is increasingly recognised. For example, among other initiatives, the American Medical Association Accelerating Change in Medical Education Consortium aims to create flexible, individualised learning plans to optimise the healthcare learning environment (Andrews et al. 2021). This pilot and the NSW PDP process have similar aims. The NASSS analysis of the pilot data showed that adopting and embedding a supported mandatory PDP process for HNSP doctors in NSW, while acceptable and feasible, will be complex. This was expected, given the diversity of the workforce, clinical workplaces and the overarching health system. The complexity implies that the process rollout is likely to be unpredictable in its adoption, dynamic and emergent (Greenhalgh et al. 2017). Twenty-three recommendations for statewide implementation of the PDP process arising from the pilot have been discussed with NSW Health. We have recommended that the supervisor supported PDP process be mandatory. While professional development may be more sustained if PDPs are voluntary (Smith & Tillema 2001), PDPs are becoming mandatory in Australia in 2023 in any event. If they are developed with supervisor input that addresses learners' needs, they are less likely to be a tick box exercise, as many mandatory activities become (Macdougall, Epstein & Highet 2017).

As with other transitions in healthcare, the addition of a PDP process for HNSP doctors needs to be supported by a timely, statewide orientation that includes the rationale for its implementation. Online resources for HNSP doctors and supervisors are being developed. These explain the process, show what a PDP should consist of and demonstrate examples of content. The doctors should be supported to develop their learning goals with supervisors' input. The focus of the process needs to be on the individual doctor with facilitation tailored to their preferences (Jennings 2007), career goals and experience. The required support level depends on each doctor's experience with PDPs as medical students or prevocational doctors (PGY1 and PGY2) and their comfort with feedback dialogues. Many interviewees stated they had little to no experience setting learning goals and variable experience in engaging with feedback. It is critical to avoid the tensions that have arisen with the portfolio-based assessment of students (Oudkerk Pool et al. 2020). The PDP process should be viewed as a supportive activity for learning rather than an assessment so that doctors are honest about their strengths and areas for improvement. Engaged doctors should require less supervisory input as they progress, particularly if the PDP process enhances their agency and ability to take advantage of a range of work-based learning opportunities (Watling et al. 2021). This would give supervisors more time to motivate non-adopters. Consideration will need to be given to the time and resources required to train supervisors. An ePortfolio may be advisable to develop and share PDPs, though these have cost and security implications. In medical training, PDP portfolios may have contrasting purposes in assessment and support (van der Gulden et al. 2022). In this process, the ePortfolio would have two goals related to support: (1) monitoring and planning doctor development, and (2) stimulating reflection (Driessen & van Tartwijk 2018). The lack of a suitable technological platform that is supported statewide complicates the adoption of the PDP process.

The process needs to be adaptable to local hospital and district circumstances, such as workforce issues (particularly in rural and remote areas), supervisory capacity and experience, individual HNSP doctors' needs and availability of resources (hours and support). Significant time and effort across the organisation are required to set up and document meetings and find suitable times to meet. These factors may be mitigated by paid additional administration support as provided in the current pilot. Governance is also needed to ensure that doctors meet the PDP process requirements.

STRENGTHS AND LIMITATIONS

A strength of the pilot evaluation is using the NASSS framework to inform data collection and analysis. The study did not solely look at outcomes. It also examined the development and implementation process and implications for the wider rollout of the PDP process.

The pilot was undertaken at four sites. Almost half of the HNSP doctors involved (excluding those at Site D) and 13 supervisors were interviewed. The study also analysed PDPs and collected data on administration and supervision time. This resulted in a sizeable amount of data to apply the NASSS framework, make recommendations for the rollout and enhance sustainability. However, due to the short length of the pilot (13 weeks), we were unable to explore longer-term outcomes in terms of benefits for the HNSP doctors and the health system.

The HNSP doctors and supervisors were recruited to the pilot following successful expressions of interest by their LHDs. Those doctors who consented to be interviewed were more likely to be positive about the process than those who did not or the small number who withdrew or did not engage with the PDP process. A late start and slow uptake hampered the process at one site, but this experience informed the evaluation.

The findings of this evaluation may not be transferable to all NSW Health facilities or other states and territories. However, within the applied framework, they capture the complicated and complex nature of the HNSP condition. This has informed significant recommendations that recognise those complexities and how they may be mitigated.

Given the short time frame, we did not conduct a cost evaluation or explore cost-effectiveness, apart from collecting data about supervisor and administrative time required during the pilot. There need to be more cost-related studies in professional health education, including continuing professional development (Cook, Wilkinson & Foo 2022). This aspect of the process will be more closely examined when it is implemented statewide.

Conclusion

The evaluation of the PDP process for doctors in unaccredited positions indicates that the process is acceptable and feasible. However, it must also be capable of adapting to local circumstances, including workforce, supervisory capacity and experience, individual doctor needs and available resources.

Note: PDPs are mandatory in Australia for all doctors (with a few exceptions). At the time of this publication, this was dependent on doctors having a CPD home, and there were no suitable CPD homes for doctors in unaccredited positions. Therefore, the requirement to have a PDP will not be enforced until 2024 for this group.

Acknowledgements

We want to thank all the doctors, supervisors and administrative support staff who participated in this pilot.

Conflict of interest

The authors declare no conflicts of interest.

Funding

This pilot evaluation was funded by NSW Health and carried out by the authors as part of their regular remunerated positions with HETI.

References

AMA NSW 2019, Protection for doctors-in-training, AMA, viewed 25 October 2022, https://www.amansw.com.au/protections-for-doctors-in-training/

Andrews, JS, Lomis, KD, Richardson, JA, Hammoud, MM & Skochelak, SE 2021, 'Expanding innovation from undergraduate to graduate medical education: A path of continuous professional development', Medical Teacher , vol. 43, sup. 2, pp. S49-S55. https://doi.org/10.1080/0142159X.2021.1935835

Australian Government Department of Health 2021, National Medical Workforce Strategy 2021-2031 . Australian Government Department of Health.

Beausaert, S, Segers, MR, van der Rijt, J & Gijselaers, WH 2011, 'The use of personal development plans (PDPs) in the workplace: A literature review', in P van den Bossche, W Gijselaers & R Milter (eds), Building learning experiences in a changing world ,Springer, Dordrecht.

Challis, M 2000, 'AMEE medical education guide no. 19: Personal learning plans', Medical Teacher, vol. 22, no. 3, pp. 225-236.

Cook, DA, Wilkinson, JM & Foo, J 2022, 'Quality of cost evaluations of physician continuous professional development: Systematic review of reporting and methods', Perspectives on Medical Education, vol. 11, no. 1. https://doi.org/10.1007/s40037-022-00705-z

Doran, GT 1981, 'There's a S.M.A.R.T. way to write management's goals and objectives', Management Review, vol. 70, no. 11, pp.35-36.

Driessen, E & van Tartwijk, J 2018, 'Portfolios in personal and professional development', in T Swanwick, B O'Brien & K Forrest (eds), Understanding medical education, 2nd edn, Wiley, Oxford.

Greenhalgh, T & Papoutsi C 2019, 'Spreading and scaling up innovation and improvement', British Medical Journal, vol. 365. 12068. https://doi.org/10.1136/bmj.l2068

Greenhalgh, T, Wherton, J, Papoutsi, C, Lynch, L., Hughes, G., A'Court, C, Hinder, S, Fahy, N, Procter, R & Shaw, S 2017, 'Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread and sustainability of health and care technologies', Journal of Medical Internet Research, vol. 19, no. 11. https://doi.org/10.2196/jmir.8775

Jennings, SF 2007, 'Personal development plans and self-directed learning for healthcare professionals: Are they evidence-based?' Postgraduate Medical Journal , vol. 83, pp. 518-524.

Liamputtong, P 2020, Qualitative research methods, Oxford University Press, Docklands, Victoria.

Lindsay, T 2019, Time to end unaccredited registrar positions, Insight+, viewed 25 October 2022, https://insightplus.mja.com.au/2019/5/time-to-end-unaccredited-registrar-positions/

Macdougall, C, Epstein, M & Highet, L 2017, 'Continuing professional development: Putting the learner back at the centre', Archives of Disease in Children: Education and Practice , vol. 102, pp. 249-253.

McKinnell, J 2019, Sydney surgeon quits dream job after seeing 'the dark side' of the profession , ABC News, viewed 25 October 2022 https://www.abc.net.au/news/2019-02-06/sydney-surgeon-doctor-hours-unsw-yumiko-kadota-stress/10784238

Medical Board of Australia, AHPRA, 2021, Registration standard: Continuing professional development , Medical Board of Australia, Canberra.

Nilsen P 2015, 'Making sense of implementation theories, models and frameworks', Implementation Science, vol. 10, p. 53. https://doi.org/10.1186/s13012-015-0242-0

Oudkerk Pool, A, Jaarsma, ADC, Driessen, EW & Govaerts, MJB 2020, 'Student perspectives on competency-based portfolios: Does a portfolio reflect their competence development?' Perspectives on Medical Education , vol. 9, pp. 166-172. https://doi.org/10.1007/s40037-020-00571-7

Ozolins, I, Leslie, S, Greene, D & Vial, T 2014, HSP professional development process pilot , HETI, Sydney.

Smith, K & Tillema, H 2001, 'Long-term influences of portfolios on professional development', Journal of Educational Research, vol. 45, pp. 183-202.

van der Gulden, R, Timmerman, A, Muris JWM, Thoonen, BPA, Heeneman, S & Scherpbier-de-Haan, ND 2022, 'How does portfolio use affect self-regulated learning in clinical workplace learning: What works, for whom and in what contexts?', Perspectives on Medical Education, vol. 11, pp. 247-257. https://doi.org/10.1007/s40037-022-00727-7

Watling, C, Ginsburg, S, LaDonna, K, Lingard, L & Field, E 2021, 'Going against the grain: An exploration of agency in medical learning', Medical Education , vol. 55, pp. 942-950. https://doi.org/10.1111/medu.14532