Learn more

How it works

Transform your enterprise with the scalable mindsets, skills, & behavior change that drive performance.

Explore how BetterUp connects to your core business systems.

We pair AI with the latest in human-centered coaching to drive powerful, lasting learning and behavior change.

Build leaders that accelerate team performance and engagement.

Unlock performance potential at scale with AI-powered curated growth journeys.

Build resilience, well-being and agility to drive performance across your entire enterprise.

Transform your business, starting with your sales leaders.

Unlock business impact from the top with executive coaching.

Foster a culture of inclusion and belonging.

Accelerate the performance and potential of your agencies and employees.

See how innovative organizations use BetterUp to build a thriving workforce.

Discover how BetterUp measurably impacts key business outcomes for organizations like yours.

A demo is the first step to transforming your business. Meet with us to develop a plan for attaining your goals.

Request a demo

  • For Individuals

Best practices, research, and tools to fuel individual and business growth.

View on-demand BetterUp events and learn about upcoming live discussions.

The latest insights and ideas for building a high-performing workplace.

  • BetterUp Briefing

The online magazine that helps you understand tomorrow's workforce trends, today.

Innovative research featured in peer-reviewed journals, press, and more.

Founded in 2022 to deepen the understanding of the intersection of well-being, purpose, and performance

We're on a mission to help everyone live with clarity, purpose, and passion.

Join us and create impactful change.

Read the buzz about BetterUp.

Meet the leadership that's passionate about empowering your workforce.

For Business

31 examples of problem solving performance review phrases

Understand Yourself Better:

Big 5 Personality Test

Find my Coach

Jump to section

You're doing great

You should think of improving

Tips to improve

Use these practical examples of phrases, sample comments, and templates for your performance review , 360-degree feedback survey, or manager appraisal.

The following examples not only relate to problem-solving but also conflict management , effective solutions, selecting the best alternatives, decision making , problem identification, analyzing effectively, and generally becoming an effective problem-solving strategist. Start using effective performance review questions to help better guide your workforce's development. 

Problem solving appraisal comments: you're doing great

  • You always maintain an effective dialogue with clients when they have technical problems. Being clear and articulate makes sure our customers' faults are attended to promptly.
  • You constantly make sure to look beyond the obvious you never stop at the first answer. You’re really good at exploring alternatives. Well done!
  • Keeping the supervisors and managers informed of status changes and requests is important. You’re really good at communicating the changes to the projects at all times. Keep it up!
  • You stay cool and collected even when things aren’t going according to plan or up in the air. This is a great trait to possess. Well done!
  • You’re excellent at giving an honest and logical analysis. Keep it up! Effectively diagnosing complex problems and reaching sustainable solutions is one of your strong points.
  • Your ability to ability to make complex systems into simple ones is truly a unique skill to possess. Well done!
  • You often identify practical solutions to every roadblock. You’re a real asset to the team! Great job.
  • You always listen actively and attentively to make sure you understand what the exact problem is and you come up with solutions in an effective manner.
  • You have an amazing ability to clearly explain options and solutions effectively and efficiently. Well done!
  • When driving projects, you can shift to other areas comfortably and easily. making sure the project runs smoothly. Great job!

problem-solving-performance-review-phrases-person-at-work-talking-to-boss

Problem solving performance review phrases: you should think of improving

  • You always seem too overwhelmed when faced with multiple problems. Try to think of ways to make problems more manageable so that they can be solved in a timely and effective manner.
  • Avoiding conflicts constantly with people is not a good idea as you will only build up personal frustration and nothing will be done to remedy the situation. Try to face people when there are problems and rectify problems when they occur.
  • Don’t allow demanding customers to rattle your cage too much. If they become too demanding, take a step back, regulate your emotions , and try to make use of online support tools to help you rectify problems these tools can help a lot!
  • It’s necessary that you learn from your past mistakes . You cannot keep making the same mistakes , as this is not beneficial to the company.
  • You tend to ask the same questions over and over again. Try to listen more attentively or take notes when colleagues are answering!
  • Providing multiple solutions in an indirect and creative approach will allow you to be more effective at problem-solving . if you struggle with this typically through viewing the problem in a new and unusual light.
  • You fail to provide staff with the appropriate amount of structure and direction. They must know the direction you wish them to go in to achieve their goals .
  • You need to be able to recognize repetitive trends to solve problems promptly.
  • You tend to have problems troubleshooting even the most basic of questions. As a problem solver and customer support person, it’s imperative that you can answer these questions easily.
  • Read through your training manual and make sure you fully understand it before attempting questions again.

problem-solving-performance-review-phrases-person-talking-at-work

Performance review tips to improve problem solving

  • Try to complain less about problems and come up with solutions to the problems more often. Complaining is not beneficial to progression and innovation.
  • As a problem solver, it’s important to be able to handle multiple priorities under short deadlines.
  • You need to be able to effectively distinguish between the cause and the symptoms of problems to solve them in an efficient and timely manner.
  • Try to anticipate problems in advance before they become major roadblocks down the road.
  • Try to view obstacles as opportunities to learn and thrive at the challenge of solving the problem.
  • Remember to prioritize problems according to their degree of urgency. It's important that you spend the majority of your time on urgent tasks over menial ones.
  • When putting plans into place, stick to them and make sure they are completed.
  • When solving problems, try to allocate appropriate levels of resources when undertaking new projects. It is important to become as efficient and as effective as possible.
  • Try to learn to pace yourself when solving problems to avoid burnout . You’re a great asset to the team and we cannot afford to lose at this point.
  • Meeting regularly with your staff to review results is vital to the problem-solving process.
  • Staff that has regular check-ins understand what it is that is required of them, what they are currently achieving, and areas they may need to improve. Try to hold one-on-one meetings every week.

New call-to-action

Madeline Miles

Madeline is a writer, communicator, and storyteller who is passionate about using words to help drive positive change. She holds a bachelor's in English Creative Writing and Communication Studies and lives in Denver, Colorado. In her spare time, she's usually somewhere outside (preferably in the mountains) — and enjoys poetry and fiction.

25 performance review questions (and how to use them)

How a performance review template improves the feedback process, 10 performance review tips to drastically move the needle, 37 innovation and creativity appraisal comments, 6 surefire ways to reach optimal peak performance, leading for purpose and performance: insights from the collaborative, 18 questions to ask in a performance self-evaluation, agile performance management: how to improve an agile team, 5 tactics for managing managers effectively — and why it matters, similar articles, 10 problem-solving strategies to turn challenges on their head, teamwork skills self-appraisal comments: 40 example phrases, your complete guide to self-assessments (with examples), 30 communication feedback examples, 30 customer service review examples to develop your team, 15 tips for your end-of-year reviews, 8 creative solutions to your most challenging problems, stay connected with betterup, get our newsletter, event invites, plus product insights and research..

3100 E 5th Street, Suite 350 Austin, TX 78702

  • Platform Overview
  • Integrations
  • Powered by AI
  • BetterUp Lead
  • BetterUp Manage™
  • BetterUp Care™
  • Sales Performance
  • Diversity & Inclusion
  • Case Studies
  • Why BetterUp?
  • Career Coaching
  • Communication Coaching
  • Life Coaching
  • News and Press
  • Leadership Team
  • Become a BetterUp Coach
  • BetterUp Labs
  • Center for Purpose & Performance
  • What is coaching?
  • Leadership Training
  • Business Coaching
  • Contact Support
  • Contact Sales
  • Privacy Policy
  • Acceptable Use Policy
  • Trust & Security
  • Cookie Preferences

35 problem-solving techniques and methods for solving complex problems

Problem solving workshop

Design your next session with SessionLab

Join the 150,000+ facilitators 
using SessionLab.

Recommended Articles

A step-by-step guide to planning a workshop, how to create an unforgettable training session in 8 simple steps, 47 useful online tools for workshop planning and meeting facilitation.

All teams and organizations encounter challenges as they grow. There are problems that might occur for teams when it comes to miscommunication or resolving business-critical issues . You may face challenges around growth , design , user engagement, and even team culture and happiness. In short, problem-solving techniques should be part of every team’s skillset.

Problem-solving methods are primarily designed to help a group or team through a process of first identifying problems and challenges , ideating possible solutions , and then evaluating the most suitable .

Finding effective solutions to complex problems isn’t easy, but by using the right process and techniques, you can help your team be more efficient in the process.

So how do you develop strategies that are engaging, and empower your team to solve problems effectively?

In this blog post, we share a series of problem-solving tools you can use in your next workshop or team meeting. You’ll also find some tips for facilitating the process and how to enable others to solve complex problems.

Let’s get started! 

How do you identify problems?

How do you identify the right solution.

  • Tips for more effective problem-solving

Complete problem-solving methods

  • Problem-solving techniques to identify and analyze problems
  • Problem-solving techniques for developing solutions

Problem-solving warm-up activities

Closing activities for a problem-solving process.

Before you can move towards finding the right solution for a given problem, you first need to identify and define the problem you wish to solve. 

Here, you want to clearly articulate what the problem is and allow your group to do the same. Remember that everyone in a group is likely to have differing perspectives and alignment is necessary in order to help the group move forward. 

Identifying a problem accurately also requires that all members of a group are able to contribute their views in an open and safe manner. It can be scary for people to stand up and contribute, especially if the problems or challenges are emotive or personal in nature. Be sure to try and create a psychologically safe space for these kinds of discussions.

Remember that problem analysis and further discussion are also important. Not taking the time to fully analyze and discuss a challenge can result in the development of solutions that are not fit for purpose or do not address the underlying issue.

Successfully identifying and then analyzing a problem means facilitating a group through activities designed to help them clearly and honestly articulate their thoughts and produce usable insight.

With this data, you might then produce a problem statement that clearly describes the problem you wish to be addressed and also state the goal of any process you undertake to tackle this issue.  

Finding solutions is the end goal of any process. Complex organizational challenges can only be solved with an appropriate solution but discovering them requires using the right problem-solving tool.

After you’ve explored a problem and discussed ideas, you need to help a team discuss and choose the right solution. Consensus tools and methods such as those below help a group explore possible solutions before then voting for the best. They’re a great way to tap into the collective intelligence of the group for great results!

Remember that the process is often iterative. Great problem solvers often roadtest a viable solution in a measured way to see what works too. While you might not get the right solution on your first try, the methods below help teams land on the most likely to succeed solution while also holding space for improvement.

Every effective problem solving process begins with an agenda . A well-structured workshop is one of the best methods for successfully guiding a group from exploring a problem to implementing a solution.

In SessionLab, it’s easy to go from an idea to a complete agenda . Start by dragging and dropping your core problem solving activities into place . Add timings, breaks and necessary materials before sharing your agenda with your colleagues.

The resulting agenda will be your guide to an effective and productive problem solving session that will also help you stay organized on the day!

problem solving for evaluation

Tips for more effective problem solving

Problem-solving activities are only one part of the puzzle. While a great method can help unlock your team’s ability to solve problems, without a thoughtful approach and strong facilitation the solutions may not be fit for purpose.

Let’s take a look at some problem-solving tips you can apply to any process to help it be a success!

Clearly define the problem

Jumping straight to solutions can be tempting, though without first clearly articulating a problem, the solution might not be the right one. Many of the problem-solving activities below include sections where the problem is explored and clearly defined before moving on.

This is a vital part of the problem-solving process and taking the time to fully define an issue can save time and effort later. A clear definition helps identify irrelevant information and it also ensures that your team sets off on the right track.

Don’t jump to conclusions

It’s easy for groups to exhibit cognitive bias or have preconceived ideas about both problems and potential solutions. Be sure to back up any problem statements or potential solutions with facts, research, and adequate forethought.

The best techniques ask participants to be methodical and challenge preconceived notions. Make sure you give the group enough time and space to collect relevant information and consider the problem in a new way. By approaching the process with a clear, rational mindset, you’ll often find that better solutions are more forthcoming.  

Try different approaches  

Problems come in all shapes and sizes and so too should the methods you use to solve them. If you find that one approach isn’t yielding results and your team isn’t finding different solutions, try mixing it up. You’ll be surprised at how using a new creative activity can unblock your team and generate great solutions.

Don’t take it personally 

Depending on the nature of your team or organizational problems, it’s easy for conversations to get heated. While it’s good for participants to be engaged in the discussions, ensure that emotions don’t run too high and that blame isn’t thrown around while finding solutions.

You’re all in it together, and even if your team or area is seeing problems, that isn’t necessarily a disparagement of you personally. Using facilitation skills to manage group dynamics is one effective method of helping conversations be more constructive.

Get the right people in the room

Your problem-solving method is often only as effective as the group using it. Getting the right people on the job and managing the number of people present is important too!

If the group is too small, you may not get enough different perspectives to effectively solve a problem. If the group is too large, you can go round and round during the ideation stages.

Creating the right group makeup is also important in ensuring you have the necessary expertise and skillset to both identify and follow up on potential solutions. Carefully consider who to include at each stage to help ensure your problem-solving method is followed and positioned for success.

Document everything

The best solutions can take refinement, iteration, and reflection to come out. Get into a habit of documenting your process in order to keep all the learnings from the session and to allow ideas to mature and develop. Many of the methods below involve the creation of documents or shared resources. Be sure to keep and share these so everyone can benefit from the work done!

Bring a facilitator 

Facilitation is all about making group processes easier. With a subject as potentially emotive and important as problem-solving, having an impartial third party in the form of a facilitator can make all the difference in finding great solutions and keeping the process moving. Consider bringing a facilitator to your problem-solving session to get better results and generate meaningful solutions!

Develop your problem-solving skills

It takes time and practice to be an effective problem solver. While some roles or participants might more naturally gravitate towards problem-solving, it can take development and planning to help everyone create better solutions.

You might develop a training program, run a problem-solving workshop or simply ask your team to practice using the techniques below. Check out our post on problem-solving skills to see how you and your group can develop the right mental process and be more resilient to issues too!

Design a great agenda

Workshops are a great format for solving problems. With the right approach, you can focus a group and help them find the solutions to their own problems. But designing a process can be time-consuming and finding the right activities can be difficult.

Check out our workshop planning guide to level-up your agenda design and start running more effective workshops. Need inspiration? Check out templates designed by expert facilitators to help you kickstart your process!

In this section, we’ll look at in-depth problem-solving methods that provide a complete end-to-end process for developing effective solutions. These will help guide your team from the discovery and definition of a problem through to delivering the right solution.

If you’re looking for an all-encompassing method or problem-solving model, these processes are a great place to start. They’ll ask your team to challenge preconceived ideas and adopt a mindset for solving problems more effectively.

  • Six Thinking Hats
  • Lightning Decision Jam
  • Problem Definition Process
  • Discovery & Action Dialogue
Design Sprint 2.0
  • Open Space Technology

1. Six Thinking Hats

Individual approaches to solving a problem can be very different based on what team or role an individual holds. It can be easy for existing biases or perspectives to find their way into the mix, or for internal politics to direct a conversation.

Six Thinking Hats is a classic method for identifying the problems that need to be solved and enables your team to consider them from different angles, whether that is by focusing on facts and data, creative solutions, or by considering why a particular solution might not work.

Like all problem-solving frameworks, Six Thinking Hats is effective at helping teams remove roadblocks from a conversation or discussion and come to terms with all the aspects necessary to solve complex problems.

2. Lightning Decision Jam

Featured courtesy of Jonathan Courtney of AJ&Smart Berlin, Lightning Decision Jam is one of those strategies that should be in every facilitation toolbox. Exploring problems and finding solutions is often creative in nature, though as with any creative process, there is the potential to lose focus and get lost.

Unstructured discussions might get you there in the end, but it’s much more effective to use a method that creates a clear process and team focus.

In Lightning Decision Jam, participants are invited to begin by writing challenges, concerns, or mistakes on post-its without discussing them before then being invited by the moderator to present them to the group.

From there, the team vote on which problems to solve and are guided through steps that will allow them to reframe those problems, create solutions and then decide what to execute on. 

By deciding the problems that need to be solved as a team before moving on, this group process is great for ensuring the whole team is aligned and can take ownership over the next stages. 

Lightning Decision Jam (LDJ)   #action   #decision making   #problem solving   #issue analysis   #innovation   #design   #remote-friendly   The problem with anything that requires creative thinking is that it’s easy to get lost—lose focus and fall into the trap of having useless, open-ended, unstructured discussions. Here’s the most effective solution I’ve found: Replace all open, unstructured discussion with a clear process. What to use this exercise for: Anything which requires a group of people to make decisions, solve problems or discuss challenges. It’s always good to frame an LDJ session with a broad topic, here are some examples: The conversion flow of our checkout Our internal design process How we organise events Keeping up with our competition Improving sales flow

3. Problem Definition Process

While problems can be complex, the problem-solving methods you use to identify and solve those problems can often be simple in design. 

By taking the time to truly identify and define a problem before asking the group to reframe the challenge as an opportunity, this method is a great way to enable change.

Begin by identifying a focus question and exploring the ways in which it manifests before splitting into five teams who will each consider the problem using a different method: escape, reversal, exaggeration, distortion or wishful. Teams develop a problem objective and create ideas in line with their method before then feeding them back to the group.

This method is great for enabling in-depth discussions while also creating space for finding creative solutions too!

Problem Definition   #problem solving   #idea generation   #creativity   #online   #remote-friendly   A problem solving technique to define a problem, challenge or opportunity and to generate ideas.

4. The 5 Whys 

Sometimes, a group needs to go further with their strategies and analyze the root cause at the heart of organizational issues. An RCA or root cause analysis is the process of identifying what is at the heart of business problems or recurring challenges. 

The 5 Whys is a simple and effective method of helping a group go find the root cause of any problem or challenge and conduct analysis that will deliver results. 

By beginning with the creation of a problem statement and going through five stages to refine it, The 5 Whys provides everything you need to truly discover the cause of an issue.

The 5 Whys   #hyperisland   #innovation   This simple and powerful method is useful for getting to the core of a problem or challenge. As the title suggests, the group defines a problems, then asks the question “why” five times, often using the resulting explanation as a starting point for creative problem solving.

5. World Cafe

World Cafe is a simple but powerful facilitation technique to help bigger groups to focus their energy and attention on solving complex problems.

World Cafe enables this approach by creating a relaxed atmosphere where participants are able to self-organize and explore topics relevant and important to them which are themed around a central problem-solving purpose. Create the right atmosphere by modeling your space after a cafe and after guiding the group through the method, let them take the lead!

Making problem-solving a part of your organization’s culture in the long term can be a difficult undertaking. More approachable formats like World Cafe can be especially effective in bringing people unfamiliar with workshops into the fold. 

World Cafe   #hyperisland   #innovation   #issue analysis   World Café is a simple yet powerful method, originated by Juanita Brown, for enabling meaningful conversations driven completely by participants and the topics that are relevant and important to them. Facilitators create a cafe-style space and provide simple guidelines. Participants then self-organize and explore a set of relevant topics or questions for conversation.

6. Discovery & Action Dialogue (DAD)

One of the best approaches is to create a safe space for a group to share and discover practices and behaviors that can help them find their own solutions.

With DAD, you can help a group choose which problems they wish to solve and which approaches they will take to do so. It’s great at helping remove resistance to change and can help get buy-in at every level too!

This process of enabling frontline ownership is great in ensuring follow-through and is one of the methods you will want in your toolbox as a facilitator.

Discovery & Action Dialogue (DAD)   #idea generation   #liberating structures   #action   #issue analysis   #remote-friendly   DADs make it easy for a group or community to discover practices and behaviors that enable some individuals (without access to special resources and facing the same constraints) to find better solutions than their peers to common problems. These are called positive deviant (PD) behaviors and practices. DADs make it possible for people in the group, unit, or community to discover by themselves these PD practices. DADs also create favorable conditions for stimulating participants’ creativity in spaces where they can feel safe to invent new and more effective practices. Resistance to change evaporates as participants are unleashed to choose freely which practices they will adopt or try and which problems they will tackle. DADs make it possible to achieve frontline ownership of solutions.

7. Design Sprint 2.0

Want to see how a team can solve big problems and move forward with prototyping and testing solutions in a few days? The Design Sprint 2.0 template from Jake Knapp, author of Sprint, is a complete agenda for a with proven results.

Developing the right agenda can involve difficult but necessary planning. Ensuring all the correct steps are followed can also be stressful or time-consuming depending on your level of experience.

Use this complete 4-day workshop template if you are finding there is no obvious solution to your challenge and want to focus your team around a specific problem that might require a shortcut to launching a minimum viable product or waiting for the organization-wide implementation of a solution.

8. Open space technology

Open space technology- developed by Harrison Owen – creates a space where large groups are invited to take ownership of their problem solving and lead individual sessions. Open space technology is a great format when you have a great deal of expertise and insight in the room and want to allow for different takes and approaches on a particular theme or problem you need to be solved.

Start by bringing your participants together to align around a central theme and focus their efforts. Explain the ground rules to help guide the problem-solving process and then invite members to identify any issue connecting to the central theme that they are interested in and are prepared to take responsibility for.

Once participants have decided on their approach to the core theme, they write their issue on a piece of paper, announce it to the group, pick a session time and place, and post the paper on the wall. As the wall fills up with sessions, the group is then invited to join the sessions that interest them the most and which they can contribute to, then you’re ready to begin!

Everyone joins the problem-solving group they’ve signed up to, record the discussion and if appropriate, findings can then be shared with the rest of the group afterward.

Open Space Technology   #action plan   #idea generation   #problem solving   #issue analysis   #large group   #online   #remote-friendly   Open Space is a methodology for large groups to create their agenda discerning important topics for discussion, suitable for conferences, community gatherings and whole system facilitation

Techniques to identify and analyze problems

Using a problem-solving method to help a team identify and analyze a problem can be a quick and effective addition to any workshop or meeting.

While further actions are always necessary, you can generate momentum and alignment easily, and these activities are a great place to get started.

We’ve put together this list of techniques to help you and your team with problem identification, analysis, and discussion that sets the foundation for developing effective solutions.

Let’s take a look!

  • The Creativity Dice
  • Fishbone Analysis
  • Problem Tree
  • SWOT Analysis
  • Agreement-Certainty Matrix
  • The Journalistic Six
  • LEGO Challenge
  • What, So What, Now What?
  • Journalists

Individual and group perspectives are incredibly important, but what happens if people are set in their minds and need a change of perspective in order to approach a problem more effectively?

Flip It is a method we love because it is both simple to understand and run, and allows groups to understand how their perspectives and biases are formed. 

Participants in Flip It are first invited to consider concerns, issues, or problems from a perspective of fear and write them on a flip chart. Then, the group is asked to consider those same issues from a perspective of hope and flip their understanding.  

No problem and solution is free from existing bias and by changing perspectives with Flip It, you can then develop a problem solving model quickly and effectively.

Flip It!   #gamestorming   #problem solving   #action   Often, a change in a problem or situation comes simply from a change in our perspectives. Flip It! is a quick game designed to show players that perspectives are made, not born.

10. The Creativity Dice

One of the most useful problem solving skills you can teach your team is of approaching challenges with creativity, flexibility, and openness. Games like The Creativity Dice allow teams to overcome the potential hurdle of too much linear thinking and approach the process with a sense of fun and speed. 

In The Creativity Dice, participants are organized around a topic and roll a dice to determine what they will work on for a period of 3 minutes at a time. They might roll a 3 and work on investigating factual information on the chosen topic. They might roll a 1 and work on identifying the specific goals, standards, or criteria for the session.

Encouraging rapid work and iteration while asking participants to be flexible are great skills to cultivate. Having a stage for idea incubation in this game is also important. Moments of pause can help ensure the ideas that are put forward are the most suitable. 

The Creativity Dice   #creativity   #problem solving   #thiagi   #issue analysis   Too much linear thinking is hazardous to creative problem solving. To be creative, you should approach the problem (or the opportunity) from different points of view. You should leave a thought hanging in mid-air and move to another. This skipping around prevents premature closure and lets your brain incubate one line of thought while you consciously pursue another.

11. Fishbone Analysis

Organizational or team challenges are rarely simple, and it’s important to remember that one problem can be an indication of something that goes deeper and may require further consideration to be solved.

Fishbone Analysis helps groups to dig deeper and understand the origins of a problem. It’s a great example of a root cause analysis method that is simple for everyone on a team to get their head around. 

Participants in this activity are asked to annotate a diagram of a fish, first adding the problem or issue to be worked on at the head of a fish before then brainstorming the root causes of the problem and adding them as bones on the fish. 

Using abstractions such as a diagram of a fish can really help a team break out of their regular thinking and develop a creative approach.

Fishbone Analysis   #problem solving   ##root cause analysis   #decision making   #online facilitation   A process to help identify and understand the origins of problems, issues or observations.

12. Problem Tree 

Encouraging visual thinking can be an essential part of many strategies. By simply reframing and clarifying problems, a group can move towards developing a problem solving model that works for them. 

In Problem Tree, groups are asked to first brainstorm a list of problems – these can be design problems, team problems or larger business problems – and then organize them into a hierarchy. The hierarchy could be from most important to least important or abstract to practical, though the key thing with problem solving games that involve this aspect is that your group has some way of managing and sorting all the issues that are raised.

Once you have a list of problems that need to be solved and have organized them accordingly, you’re then well-positioned for the next problem solving steps.

Problem tree   #define intentions   #create   #design   #issue analysis   A problem tree is a tool to clarify the hierarchy of problems addressed by the team within a design project; it represents high level problems or related sublevel problems.

13. SWOT Analysis

Chances are you’ve heard of the SWOT Analysis before. This problem-solving method focuses on identifying strengths, weaknesses, opportunities, and threats is a tried and tested method for both individuals and teams.

Start by creating a desired end state or outcome and bare this in mind – any process solving model is made more effective by knowing what you are moving towards. Create a quadrant made up of the four categories of a SWOT analysis and ask participants to generate ideas based on each of those quadrants.

Once you have those ideas assembled in their quadrants, cluster them together based on their affinity with other ideas. These clusters are then used to facilitate group conversations and move things forward. 

SWOT analysis   #gamestorming   #problem solving   #action   #meeting facilitation   The SWOT Analysis is a long-standing technique of looking at what we have, with respect to the desired end state, as well as what we could improve on. It gives us an opportunity to gauge approaching opportunities and dangers, and assess the seriousness of the conditions that affect our future. When we understand those conditions, we can influence what comes next.

14. Agreement-Certainty Matrix

Not every problem-solving approach is right for every challenge, and deciding on the right method for the challenge at hand is a key part of being an effective team.

The Agreement Certainty matrix helps teams align on the nature of the challenges facing them. By sorting problems from simple to chaotic, your team can understand what methods are suitable for each problem and what they can do to ensure effective results. 

If you are already using Liberating Structures techniques as part of your problem-solving strategy, the Agreement-Certainty Matrix can be an invaluable addition to your process. We’ve found it particularly if you are having issues with recurring problems in your organization and want to go deeper in understanding the root cause. 

Agreement-Certainty Matrix   #issue analysis   #liberating structures   #problem solving   You can help individuals or groups avoid the frequent mistake of trying to solve a problem with methods that are not adapted to the nature of their challenge. The combination of two questions makes it possible to easily sort challenges into four categories: simple, complicated, complex , and chaotic .  A problem is simple when it can be solved reliably with practices that are easy to duplicate.  It is complicated when experts are required to devise a sophisticated solution that will yield the desired results predictably.  A problem is complex when there are several valid ways to proceed but outcomes are not predictable in detail.  Chaotic is when the context is too turbulent to identify a path forward.  A loose analogy may be used to describe these differences: simple is like following a recipe, complicated like sending a rocket to the moon, complex like raising a child, and chaotic is like the game “Pin the Tail on the Donkey.”  The Liberating Structures Matching Matrix in Chapter 5 can be used as the first step to clarify the nature of a challenge and avoid the mismatches between problems and solutions that are frequently at the root of chronic, recurring problems.

Organizing and charting a team’s progress can be important in ensuring its success. SQUID (Sequential Question and Insight Diagram) is a great model that allows a team to effectively switch between giving questions and answers and develop the skills they need to stay on track throughout the process. 

Begin with two different colored sticky notes – one for questions and one for answers – and with your central topic (the head of the squid) on the board. Ask the group to first come up with a series of questions connected to their best guess of how to approach the topic. Ask the group to come up with answers to those questions, fix them to the board and connect them with a line. After some discussion, go back to question mode by responding to the generated answers or other points on the board.

It’s rewarding to see a diagram grow throughout the exercise, and a completed SQUID can provide a visual resource for future effort and as an example for other teams.

SQUID   #gamestorming   #project planning   #issue analysis   #problem solving   When exploring an information space, it’s important for a group to know where they are at any given time. By using SQUID, a group charts out the territory as they go and can navigate accordingly. SQUID stands for Sequential Question and Insight Diagram.

16. Speed Boat

To continue with our nautical theme, Speed Boat is a short and sweet activity that can help a team quickly identify what employees, clients or service users might have a problem with and analyze what might be standing in the way of achieving a solution.

Methods that allow for a group to make observations, have insights and obtain those eureka moments quickly are invaluable when trying to solve complex problems.

In Speed Boat, the approach is to first consider what anchors and challenges might be holding an organization (or boat) back. Bonus points if you are able to identify any sharks in the water and develop ideas that can also deal with competitors!   

Speed Boat   #gamestorming   #problem solving   #action   Speedboat is a short and sweet way to identify what your employees or clients don’t like about your product/service or what’s standing in the way of a desired goal.

17. The Journalistic Six

Some of the most effective ways of solving problems is by encouraging teams to be more inclusive and diverse in their thinking.

Based on the six key questions journalism students are taught to answer in articles and news stories, The Journalistic Six helps create teams to see the whole picture. By using who, what, when, where, why, and how to facilitate the conversation and encourage creative thinking, your team can make sure that the problem identification and problem analysis stages of the are covered exhaustively and thoughtfully. Reporter’s notebook and dictaphone optional.

The Journalistic Six – Who What When Where Why How   #idea generation   #issue analysis   #problem solving   #online   #creative thinking   #remote-friendly   A questioning method for generating, explaining, investigating ideas.

18. LEGO Challenge

Now for an activity that is a little out of the (toy) box. LEGO Serious Play is a facilitation methodology that can be used to improve creative thinking and problem-solving skills. 

The LEGO Challenge includes giving each member of the team an assignment that is hidden from the rest of the group while they create a structure without speaking.

What the LEGO challenge brings to the table is a fun working example of working with stakeholders who might not be on the same page to solve problems. Also, it’s LEGO! Who doesn’t love LEGO! 

LEGO Challenge   #hyperisland   #team   A team-building activity in which groups must work together to build a structure out of LEGO, but each individual has a secret “assignment” which makes the collaborative process more challenging. It emphasizes group communication, leadership dynamics, conflict, cooperation, patience and problem solving strategy.

19. What, So What, Now What?

If not carefully managed, the problem identification and problem analysis stages of the problem-solving process can actually create more problems and misunderstandings.

The What, So What, Now What? problem-solving activity is designed to help collect insights and move forward while also eliminating the possibility of disagreement when it comes to identifying, clarifying, and analyzing organizational or work problems. 

Facilitation is all about bringing groups together so that might work on a shared goal and the best problem-solving strategies ensure that teams are aligned in purpose, if not initially in opinion or insight.

Throughout the three steps of this game, you give everyone on a team to reflect on a problem by asking what happened, why it is important, and what actions should then be taken. 

This can be a great activity for bringing our individual perceptions about a problem or challenge and contextualizing it in a larger group setting. This is one of the most important problem-solving skills you can bring to your organization.

W³ – What, So What, Now What?   #issue analysis   #innovation   #liberating structures   You can help groups reflect on a shared experience in a way that builds understanding and spurs coordinated action while avoiding unproductive conflict. It is possible for every voice to be heard while simultaneously sifting for insights and shaping new direction. Progressing in stages makes this practical—from collecting facts about What Happened to making sense of these facts with So What and finally to what actions logically follow with Now What . The shared progression eliminates most of the misunderstandings that otherwise fuel disagreements about what to do. Voila!

20. Journalists  

Problem analysis can be one of the most important and decisive stages of all problem-solving tools. Sometimes, a team can become bogged down in the details and are unable to move forward.

Journalists is an activity that can avoid a group from getting stuck in the problem identification or problem analysis stages of the process.

In Journalists, the group is invited to draft the front page of a fictional newspaper and figure out what stories deserve to be on the cover and what headlines those stories will have. By reframing how your problems and challenges are approached, you can help a team move productively through the process and be better prepared for the steps to follow.

Journalists   #vision   #big picture   #issue analysis   #remote-friendly   This is an exercise to use when the group gets stuck in details and struggles to see the big picture. Also good for defining a vision.

Problem-solving techniques for developing solutions 

The success of any problem-solving process can be measured by the solutions it produces. After you’ve defined the issue, explored existing ideas, and ideated, it’s time to narrow down to the correct solution.

Use these problem-solving techniques when you want to help your team find consensus, compare possible solutions, and move towards taking action on a particular problem.

  • Improved Solutions
  • Four-Step Sketch
  • 15% Solutions
  • How-Now-Wow matrix
  • Impact Effort Matrix

21. Mindspin  

Brainstorming is part of the bread and butter of the problem-solving process and all problem-solving strategies benefit from getting ideas out and challenging a team to generate solutions quickly. 

With Mindspin, participants are encouraged not only to generate ideas but to do so under time constraints and by slamming down cards and passing them on. By doing multiple rounds, your team can begin with a free generation of possible solutions before moving on to developing those solutions and encouraging further ideation. 

This is one of our favorite problem-solving activities and can be great for keeping the energy up throughout the workshop. Remember the importance of helping people become engaged in the process – energizing problem-solving techniques like Mindspin can help ensure your team stays engaged and happy, even when the problems they’re coming together to solve are complex. 

MindSpin   #teampedia   #idea generation   #problem solving   #action   A fast and loud method to enhance brainstorming within a team. Since this activity has more than round ideas that are repetitive can be ruled out leaving more creative and innovative answers to the challenge.

22. Improved Solutions

After a team has successfully identified a problem and come up with a few solutions, it can be tempting to call the work of the problem-solving process complete. That said, the first solution is not necessarily the best, and by including a further review and reflection activity into your problem-solving model, you can ensure your group reaches the best possible result. 

One of a number of problem-solving games from Thiagi Group, Improved Solutions helps you go the extra mile and develop suggested solutions with close consideration and peer review. By supporting the discussion of several problems at once and by shifting team roles throughout, this problem-solving technique is a dynamic way of finding the best solution. 

Improved Solutions   #creativity   #thiagi   #problem solving   #action   #team   You can improve any solution by objectively reviewing its strengths and weaknesses and making suitable adjustments. In this creativity framegame, you improve the solutions to several problems. To maintain objective detachment, you deal with a different problem during each of six rounds and assume different roles (problem owner, consultant, basher, booster, enhancer, and evaluator) during each round. At the conclusion of the activity, each player ends up with two solutions to her problem.

23. Four Step Sketch

Creative thinking and visual ideation does not need to be confined to the opening stages of your problem-solving strategies. Exercises that include sketching and prototyping on paper can be effective at the solution finding and development stage of the process, and can be great for keeping a team engaged. 

By going from simple notes to a crazy 8s round that involves rapidly sketching 8 variations on their ideas before then producing a final solution sketch, the group is able to iterate quickly and visually. Problem-solving techniques like Four-Step Sketch are great if you have a group of different thinkers and want to change things up from a more textual or discussion-based approach.

Four-Step Sketch   #design sprint   #innovation   #idea generation   #remote-friendly   The four-step sketch is an exercise that helps people to create well-formed concepts through a structured process that includes: Review key information Start design work on paper,  Consider multiple variations , Create a detailed solution . This exercise is preceded by a set of other activities allowing the group to clarify the challenge they want to solve. See how the Four Step Sketch exercise fits into a Design Sprint

24. 15% Solutions

Some problems are simpler than others and with the right problem-solving activities, you can empower people to take immediate actions that can help create organizational change. 

Part of the liberating structures toolkit, 15% solutions is a problem-solving technique that focuses on finding and implementing solutions quickly. A process of iterating and making small changes quickly can help generate momentum and an appetite for solving complex problems.

Problem-solving strategies can live and die on whether people are onboard. Getting some quick wins is a great way of getting people behind the process.   

It can be extremely empowering for a team to realize that problem-solving techniques can be deployed quickly and easily and delineate between things they can positively impact and those things they cannot change. 

15% Solutions   #action   #liberating structures   #remote-friendly   You can reveal the actions, however small, that everyone can do immediately. At a minimum, these will create momentum, and that may make a BIG difference.  15% Solutions show that there is no reason to wait around, feel powerless, or fearful. They help people pick it up a level. They get individuals and the group to focus on what is within their discretion instead of what they cannot change.  With a very simple question, you can flip the conversation to what can be done and find solutions to big problems that are often distributed widely in places not known in advance. Shifting a few grains of sand may trigger a landslide and change the whole landscape.

25. How-Now-Wow Matrix

The problem-solving process is often creative, as complex problems usually require a change of thinking and creative response in order to find the best solutions. While it’s common for the first stages to encourage creative thinking, groups can often gravitate to familiar solutions when it comes to the end of the process. 

When selecting solutions, you don’t want to lose your creative energy! The How-Now-Wow Matrix from Gamestorming is a great problem-solving activity that enables a group to stay creative and think out of the box when it comes to selecting the right solution for a given problem.

Problem-solving techniques that encourage creative thinking and the ideation and selection of new solutions can be the most effective in organisational change. Give the How-Now-Wow Matrix a go, and not just for how pleasant it is to say out loud. 

How-Now-Wow Matrix   #gamestorming   #idea generation   #remote-friendly   When people want to develop new ideas, they most often think out of the box in the brainstorming or divergent phase. However, when it comes to convergence, people often end up picking ideas that are most familiar to them. This is called a ‘creative paradox’ or a ‘creadox’. The How-Now-Wow matrix is an idea selection tool that breaks the creadox by forcing people to weigh each idea on 2 parameters.

26. Impact and Effort Matrix

All problem-solving techniques hope to not only find solutions to a given problem or challenge but to find the best solution. When it comes to finding a solution, groups are invited to put on their decision-making hats and really think about how a proposed idea would work in practice. 

The Impact and Effort Matrix is one of the problem-solving techniques that fall into this camp, empowering participants to first generate ideas and then categorize them into a 2×2 matrix based on impact and effort.

Activities that invite critical thinking while remaining simple are invaluable. Use the Impact and Effort Matrix to move from ideation and towards evaluating potential solutions before then committing to them. 

Impact and Effort Matrix   #gamestorming   #decision making   #action   #remote-friendly   In this decision-making exercise, possible actions are mapped based on two factors: effort required to implement and potential impact. Categorizing ideas along these lines is a useful technique in decision making, as it obliges contributors to balance and evaluate suggested actions before committing to them.

27. Dotmocracy

If you’ve followed each of the problem-solving steps with your group successfully, you should move towards the end of your process with heaps of possible solutions developed with a specific problem in mind. But how do you help a group go from ideation to putting a solution into action? 

Dotmocracy – or Dot Voting -is a tried and tested method of helping a team in the problem-solving process make decisions and put actions in place with a degree of oversight and consensus. 

One of the problem-solving techniques that should be in every facilitator’s toolbox, Dot Voting is fast and effective and can help identify the most popular and best solutions and help bring a group to a decision effectively. 

Dotmocracy   #action   #decision making   #group prioritization   #hyperisland   #remote-friendly   Dotmocracy is a simple method for group prioritization or decision-making. It is not an activity on its own, but a method to use in processes where prioritization or decision-making is the aim. The method supports a group to quickly see which options are most popular or relevant. The options or ideas are written on post-its and stuck up on a wall for the whole group to see. Each person votes for the options they think are the strongest, and that information is used to inform a decision.

All facilitators know that warm-ups and icebreakers are useful for any workshop or group process. Problem-solving workshops are no different.

Use these problem-solving techniques to warm up a group and prepare them for the rest of the process. Activating your group by tapping into some of the top problem-solving skills can be one of the best ways to see great outcomes from your session.

  • Check-in/Check-out
  • Doodling Together
  • Show and Tell
  • Constellations
  • Draw a Tree

28. Check-in / Check-out

Solid processes are planned from beginning to end, and the best facilitators know that setting the tone and establishing a safe, open environment can be integral to a successful problem-solving process.

Check-in / Check-out is a great way to begin and/or bookend a problem-solving workshop. Checking in to a session emphasizes that everyone will be seen, heard, and expected to contribute. 

If you are running a series of meetings, setting a consistent pattern of checking in and checking out can really help your team get into a groove. We recommend this opening-closing activity for small to medium-sized groups though it can work with large groups if they’re disciplined!

Check-in / Check-out   #team   #opening   #closing   #hyperisland   #remote-friendly   Either checking-in or checking-out is a simple way for a team to open or close a process, symbolically and in a collaborative way. Checking-in/out invites each member in a group to be present, seen and heard, and to express a reflection or a feeling. Checking-in emphasizes presence, focus and group commitment; checking-out emphasizes reflection and symbolic closure.

29. Doodling Together  

Thinking creatively and not being afraid to make suggestions are important problem-solving skills for any group or team, and warming up by encouraging these behaviors is a great way to start. 

Doodling Together is one of our favorite creative ice breaker games – it’s quick, effective, and fun and can make all following problem-solving steps easier by encouraging a group to collaborate visually. By passing cards and adding additional items as they go, the workshop group gets into a groove of co-creation and idea development that is crucial to finding solutions to problems. 

Doodling Together   #collaboration   #creativity   #teamwork   #fun   #team   #visual methods   #energiser   #icebreaker   #remote-friendly   Create wild, weird and often funny postcards together & establish a group’s creative confidence.

30. Show and Tell

You might remember some version of Show and Tell from being a kid in school and it’s a great problem-solving activity to kick off a session.

Asking participants to prepare a little something before a workshop by bringing an object for show and tell can help them warm up before the session has even begun! Games that include a physical object can also help encourage early engagement before moving onto more big-picture thinking.

By asking your participants to tell stories about why they chose to bring a particular item to the group, you can help teams see things from new perspectives and see both differences and similarities in the way they approach a topic. Great groundwork for approaching a problem-solving process as a team! 

Show and Tell   #gamestorming   #action   #opening   #meeting facilitation   Show and Tell taps into the power of metaphors to reveal players’ underlying assumptions and associations around a topic The aim of the game is to get a deeper understanding of stakeholders’ perspectives on anything—a new project, an organizational restructuring, a shift in the company’s vision or team dynamic.

31. Constellations

Who doesn’t love stars? Constellations is a great warm-up activity for any workshop as it gets people up off their feet, energized, and ready to engage in new ways with established topics. It’s also great for showing existing beliefs, biases, and patterns that can come into play as part of your session.

Using warm-up games that help build trust and connection while also allowing for non-verbal responses can be great for easing people into the problem-solving process and encouraging engagement from everyone in the group. Constellations is great in large spaces that allow for movement and is definitely a practical exercise to allow the group to see patterns that are otherwise invisible. 

Constellations   #trust   #connection   #opening   #coaching   #patterns   #system   Individuals express their response to a statement or idea by standing closer or further from a central object. Used with teams to reveal system, hidden patterns, perspectives.

32. Draw a Tree

Problem-solving games that help raise group awareness through a central, unifying metaphor can be effective ways to warm-up a group in any problem-solving model.

Draw a Tree is a simple warm-up activity you can use in any group and which can provide a quick jolt of energy. Start by asking your participants to draw a tree in just 45 seconds – they can choose whether it will be abstract or realistic. 

Once the timer is up, ask the group how many people included the roots of the tree and use this as a means to discuss how we can ignore important parts of any system simply because they are not visible.

All problem-solving strategies are made more effective by thinking of problems critically and by exposing things that may not normally come to light. Warm-up games like Draw a Tree are great in that they quickly demonstrate some key problem-solving skills in an accessible and effective way.

Draw a Tree   #thiagi   #opening   #perspectives   #remote-friendly   With this game you can raise awarness about being more mindful, and aware of the environment we live in.

Each step of the problem-solving workshop benefits from an intelligent deployment of activities, games, and techniques. Bringing your session to an effective close helps ensure that solutions are followed through on and that you also celebrate what has been achieved.

Here are some problem-solving activities you can use to effectively close a workshop or meeting and ensure the great work you’ve done can continue afterward.

  • One Breath Feedback
  • Who What When Matrix
  • Response Cards

How do I conclude a problem-solving process?

All good things must come to an end. With the bulk of the work done, it can be tempting to conclude your workshop swiftly and without a moment to debrief and align. This can be problematic in that it doesn’t allow your team to fully process the results or reflect on the process.

At the end of an effective session, your team will have gone through a process that, while productive, can be exhausting. It’s important to give your group a moment to take a breath, ensure that they are clear on future actions, and provide short feedback before leaving the space. 

The primary purpose of any problem-solving method is to generate solutions and then implement them. Be sure to take the opportunity to ensure everyone is aligned and ready to effectively implement the solutions you produced in the workshop.

Remember that every process can be improved and by giving a short moment to collect feedback in the session, you can further refine your problem-solving methods and see further success in the future too.

33. One Breath Feedback

Maintaining attention and focus during the closing stages of a problem-solving workshop can be tricky and so being concise when giving feedback can be important. It’s easy to incur “death by feedback” should some team members go on for too long sharing their perspectives in a quick feedback round. 

One Breath Feedback is a great closing activity for workshops. You give everyone an opportunity to provide feedback on what they’ve done but only in the space of a single breath. This keeps feedback short and to the point and means that everyone is encouraged to provide the most important piece of feedback to them. 

One breath feedback   #closing   #feedback   #action   This is a feedback round in just one breath that excels in maintaining attention: each participants is able to speak during just one breath … for most people that’s around 20 to 25 seconds … unless of course you’ve been a deep sea diver in which case you’ll be able to do it for longer.

34. Who What When Matrix 

Matrices feature as part of many effective problem-solving strategies and with good reason. They are easily recognizable, simple to use, and generate results.

The Who What When Matrix is a great tool to use when closing your problem-solving session by attributing a who, what and when to the actions and solutions you have decided upon. The resulting matrix is a simple, easy-to-follow way of ensuring your team can move forward. 

Great solutions can’t be enacted without action and ownership. Your problem-solving process should include a stage for allocating tasks to individuals or teams and creating a realistic timeframe for those solutions to be implemented or checked out. Use this method to keep the solution implementation process clear and simple for all involved. 

Who/What/When Matrix   #gamestorming   #action   #project planning   With Who/What/When matrix, you can connect people with clear actions they have defined and have committed to.

35. Response cards

Group discussion can comprise the bulk of most problem-solving activities and by the end of the process, you might find that your team is talked out! 

Providing a means for your team to give feedback with short written notes can ensure everyone is head and can contribute without the need to stand up and talk. Depending on the needs of the group, giving an alternative can help ensure everyone can contribute to your problem-solving model in the way that makes the most sense for them.

Response Cards is a great way to close a workshop if you are looking for a gentle warm-down and want to get some swift discussion around some of the feedback that is raised. 

Response Cards   #debriefing   #closing   #structured sharing   #questions and answers   #thiagi   #action   It can be hard to involve everyone during a closing of a session. Some might stay in the background or get unheard because of louder participants. However, with the use of Response Cards, everyone will be involved in providing feedback or clarify questions at the end of a session.

Save time and effort discovering the right solutions

A structured problem solving process is a surefire way of solving tough problems, discovering creative solutions and driving organizational change. But how can you design for successful outcomes?

With SessionLab, it’s easy to design engaging workshops that deliver results. Drag, drop and reorder blocks  to build your agenda. When you make changes or update your agenda, your session  timing   adjusts automatically , saving you time on manual adjustments.

Collaborating with stakeholders or clients? Share your agenda with a single click and collaborate in real-time. No more sending documents back and forth over email.

Explore  how to use SessionLab  to design effective problem solving workshops or  watch this five minute video  to see the planner in action!

problem solving for evaluation

Over to you

The problem-solving process can often be as complicated and multifaceted as the problems they are set-up to solve. With the right problem-solving techniques and a mix of creative exercises designed to guide discussion and generate purposeful ideas, we hope we’ve given you the tools to find the best solutions as simply and easily as possible.

Is there a problem-solving technique that you are missing here? Do you have a favorite activity or method you use when facilitating? Let us know in the comments below, we’d love to hear from you! 

' src=

thank you very much for these excellent techniques

' src=

Certainly wonderful article, very detailed. Shared!

Leave a Comment Cancel reply

Your email address will not be published. Required fields are marked *

cycle of workshop planning steps

Going from a mere idea to a workshop that delivers results for your clients can feel like a daunting task. In this piece, we will shine a light on all the work behind the scenes and help you learn how to plan a workshop from start to finish. On a good day, facilitation can feel like effortless magic, but that is mostly the result of backstage work, foresight, and a lot of careful planning. Read on to learn a step-by-step approach to breaking the process of planning a workshop into small, manageable chunks.  The flow starts with the first meeting with a client to define the purposes of a workshop.…

problem solving for evaluation

How does learning work? A clever 9-year-old once told me: “I know I am learning something new when I am surprised.” The science of adult learning tells us that, in order to learn new skills (which, unsurprisingly, is harder for adults to do than kids) grown-ups need to first get into a specific headspace.  In a business, this approach is often employed in a training session where employees learn new skills or work on professional development. But how do you ensure your training is effective? In this guide, we'll explore how to create an effective training session plan and run engaging training sessions. As team leader, project manager, or consultant,…

problem solving for evaluation

Effective online tools are a necessity for smooth and engaging virtual workshops and meetings. But how do you choose the right ones? Do you sometimes feel that the good old pen and paper or MS Office toolkit and email leaves you struggling to stay on top of managing and delivering your workshop? Fortunately, there are plenty of online tools to make your life easier when you need to facilitate a meeting and lead workshops. In this post, we’ll share our favorite online tools you can use to make your job as a facilitator easier. In fact, there are plenty of free online workshop tools and meeting facilitation software you can…

Design your next workshop with SessionLab

Join the 150,000 facilitators using SessionLab

Sign up for free

  • Bipolar Disorder
  • Therapy Center
  • When To See a Therapist
  • Types of Therapy
  • Best Online Therapy
  • Best Couples Therapy
  • Best Family Therapy
  • Managing Stress
  • Sleep and Dreaming
  • Understanding Emotions
  • Self-Improvement
  • Healthy Relationships
  • Student Resources
  • Personality Types
  • Verywell Mind Insights
  • 2023 Verywell Mind 25
  • Mental Health in the Classroom
  • Editorial Process
  • Meet Our Review Board
  • Crisis Support

Overview of the Problem-Solving Mental Process

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

problem solving for evaluation

Rachel Goldman, PhD FTOS, is a licensed psychologist, clinical assistant professor, speaker, wellness expert specializing in eating behaviors, stress management, and health behavior change.

problem solving for evaluation

  • Identify the Problem
  • Define the Problem
  • Form a Strategy
  • Organize Information
  • Allocate Resources
  • Monitor Progress
  • Evaluate the Results

Frequently Asked Questions

Problem-solving is a mental process that involves discovering, analyzing, and solving problems. The ultimate goal of problem-solving is to overcome obstacles and find a solution that best resolves the issue.

The best strategy for solving a problem depends largely on the unique situation. In some cases, people are better off learning everything they can about the issue and then using factual knowledge to come up with a solution. In other instances, creativity and insight are the best options.

It is not necessary to follow problem-solving steps sequentially, It is common to skip steps or even go back through steps multiple times until the desired solution is reached.

In order to correctly solve a problem, it is often important to follow a series of steps. Researchers sometimes refer to this as the problem-solving cycle. While this cycle is portrayed sequentially, people rarely follow a rigid series of steps to find a solution.

The following steps include developing strategies and organizing knowledge.

1. Identifying the Problem

While it may seem like an obvious step, identifying the problem is not always as simple as it sounds. In some cases, people might mistakenly identify the wrong source of a problem, which will make attempts to solve it inefficient or even useless.

Some strategies that you might use to figure out the source of a problem include :

  • Asking questions about the problem
  • Breaking the problem down into smaller pieces
  • Looking at the problem from different perspectives
  • Conducting research to figure out what relationships exist between different variables

2. Defining the Problem

After the problem has been identified, it is important to fully define the problem so that it can be solved. You can define a problem by operationally defining each aspect of the problem and setting goals for what aspects of the problem you will address

At this point, you should focus on figuring out which aspects of the problems are facts and which are opinions. State the problem clearly and identify the scope of the solution.

3. Forming a Strategy

After the problem has been identified, it is time to start brainstorming potential solutions. This step usually involves generating as many ideas as possible without judging their quality. Once several possibilities have been generated, they can be evaluated and narrowed down.

The next step is to develop a strategy to solve the problem. The approach used will vary depending upon the situation and the individual's unique preferences. Common problem-solving strategies include heuristics and algorithms.

  • Heuristics are mental shortcuts that are often based on solutions that have worked in the past. They can work well if the problem is similar to something you have encountered before and are often the best choice if you need a fast solution.
  • Algorithms are step-by-step strategies that are guaranteed to produce a correct result. While this approach is great for accuracy, it can also consume time and resources.

Heuristics are often best used when time is of the essence, while algorithms are a better choice when a decision needs to be as accurate as possible.

4. Organizing Information

Before coming up with a solution, you need to first organize the available information. What do you know about the problem? What do you not know? The more information that is available the better prepared you will be to come up with an accurate solution.

When approaching a problem, it is important to make sure that you have all the data you need. Making a decision without adequate information can lead to biased or inaccurate results.

5. Allocating Resources

Of course, we don't always have unlimited money, time, and other resources to solve a problem. Before you begin to solve a problem, you need to determine how high priority it is.

If it is an important problem, it is probably worth allocating more resources to solving it. If, however, it is a fairly unimportant problem, then you do not want to spend too much of your available resources on coming up with a solution.

At this stage, it is important to consider all of the factors that might affect the problem at hand. This includes looking at the available resources, deadlines that need to be met, and any possible risks involved in each solution. After careful evaluation, a decision can be made about which solution to pursue.

6. Monitoring Progress

After selecting a problem-solving strategy, it is time to put the plan into action and see if it works. This step might involve trying out different solutions to see which one is the most effective.

It is also important to monitor the situation after implementing a solution to ensure that the problem has been solved and that no new problems have arisen as a result of the proposed solution.

Effective problem-solvers tend to monitor their progress as they work towards a solution. If they are not making good progress toward reaching their goal, they will reevaluate their approach or look for new strategies .

7. Evaluating the Results

After a solution has been reached, it is important to evaluate the results to determine if it is the best possible solution to the problem. This evaluation might be immediate, such as checking the results of a math problem to ensure the answer is correct, or it can be delayed, such as evaluating the success of a therapy program after several months of treatment.

Once a problem has been solved, it is important to take some time to reflect on the process that was used and evaluate the results. This will help you to improve your problem-solving skills and become more efficient at solving future problems.

A Word From Verywell​

It is important to remember that there are many different problem-solving processes with different steps, and this is just one example. Problem-solving in real-world situations requires a great deal of resourcefulness, flexibility, resilience, and continuous interaction with the environment.

Get Advice From The Verywell Mind Podcast

Hosted by therapist Amy Morin, LCSW, this episode of The Verywell Mind Podcast shares how you can stop dwelling in a negative mindset.

Follow Now : Apple Podcasts / Spotify / Google Podcasts

You can become a better problem solving by:

  • Practicing brainstorming and coming up with multiple potential solutions to problems
  • Being open-minded and considering all possible options before making a decision
  • Breaking down problems into smaller, more manageable pieces
  • Asking for help when needed
  • Researching different problem-solving techniques and trying out new ones
  • Learning from mistakes and using them as opportunities to grow

It's important to communicate openly and honestly with your partner about what's going on. Try to see things from their perspective as well as your own. Work together to find a resolution that works for both of you. Be willing to compromise and accept that there may not be a perfect solution.

Take breaks if things are getting too heated, and come back to the problem when you feel calm and collected. Don't try to fix every problem on your own—consider asking a therapist or counselor for help and insight.

If you've tried everything and there doesn't seem to be a way to fix the problem, you may have to learn to accept it. This can be difficult, but try to focus on the positive aspects of your life and remember that every situation is temporary. Don't dwell on what's going wrong—instead, think about what's going right. Find support by talking to friends or family. Seek professional help if you're having trouble coping.

Davidson JE, Sternberg RJ, editors.  The Psychology of Problem Solving .  Cambridge University Press; 2003. doi:10.1017/CBO9780511615771

Sarathy V. Real world problem-solving .  Front Hum Neurosci . 2018;12:261. Published 2018 Jun 26. doi:10.3389/fnhum.2018.00261

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Teamflect Blog

100 Useful Performance Review Comments to Choose From!

problem solving for evaluation

Allow us to cut to the chase: We put together a hundred of the best performance review comments on some of the most important performance review question categories out there. Each manager has their own way of conducting performance reviews and their own performance review comments or classic performance review phrases that they like to make use of. We have no intention of messing with yours!

We wanted to put together a list of performance review comments that would serve as guiding examples for you to draft your own performance review comments around. That being said if you decide to knick these performance review comments; We promise; We won’t tell a soul!

At the end of the day, performance reviews are an extraordinary opportunity for the exchange of quality feedback, valuable insights, and just a healthy dose of anxiety. No matter which performance review comments you decide to go with and which performance review frequency you conduct your reviews with, just make sure your feedback is honest and from the heart!

Before we go any further, we should point out that no matter which performance review phrases you pick from this list, they won’t be effective if your review process is cumbersome.

Table of Contents

Are you conducting performance reviews manually?

HOW you conduct your performance reviews is equally as important as what you say in them. While we gave some pointers on streamlining performance reviews further down this list, we have special news for Microsoft Teams users.

You can conduct entire performance review cycles without ever leaving Microsoft Teams , using Teamflect’s customizable performance review templates . The best part is, that you can try this out for absolutely free! Click the button above and see just how much of a difference Teamflect makes in your performance appraisals.

Performance Review Comments Examples

1 . communication.

Performance review comments: three crumpled yellow papers on green surface surrounded by yellow lined papers

It’s no secret that communication is key to success in any job. Whether you’re writing emails, making phone calls, or having face-to-face conversations, it’s essential to be clear, concise, and confident in your communication. Of course, we all have our moments of stumbling over our words or getting tongue-tied. That’s why it’s important to work on improving our communication skills to ensure we’re always getting our message across effectively.

The feedback you give through performance reviews will be key in how the reviewee will be adjusting their communication strategies in the following review period. Here are some examples you can use:

  • “Communicates clearly and effectively, making it easy for others to understand.”
  • “Listens attentively and responds thoughtfully to others’ ideas and concerns.”
  • “Uses appropriate tone and language to convey messages, building positive relationships with team members.”
  • “Effectively summarizes information and provides relevant updates to stakeholders.”
  • “Shares feedback constructively, motivating colleagues to improve performance.”

Needs improvement:

  • “Opportunities exist for improvement in speaking with greater clarity and confidence.”
  • “Encounters challenges when expressing thoughts and ideas in a clear and effective manner.”
  • “Shows a tendency to interrupt or talk over others, which impairs communication.”
  • “Displays a challenge in receiving feedback and responding constructively to it.”
  • “Has room for improvement in active listening skills and demonstrating empathy towards colleagues.”

2. Time Management

Time management is something we all struggle with from time to time. There are only so many hours in the day, and it can be tough to juggle all our responsibilities and meet our deadlines. But fear not! With a bit of planning and some time-saving tricks up our sleeves, anyone can manage time the way Hermione Granger did in The Prisoner of Azkaban! Your performance review comments just might be the difference-maker in getting there!

  • “Consistently meets deadlines and delivers high-quality work on time.”
  • “Effectively prioritizes tasks and manages time to ensure productivity.”
  • “Demonstrates excellent organizational skills, keeping on top of multiple projects and responsibilities.”
  • “Uses time efficiently, avoiding unnecessary distractions or procrastination.”
  • “Shows flexibility in adapting to changing priorities and deadlines.”
  • “Has difficulty managing time effectively, leading to missed deadlines or rushed work.”
  • “Struggles to prioritize tasks, sometimes working on less important projects instead of urgent ones.”
  • “Tends to procrastinate, leading to work being rushed and potentially low quality.”
  • “Could benefit from better organization and planning skills to improve productivity.”
  • “May need to work on delegating tasks to others to better manage workload.”

3. Quality of Work

There are many different employee performance metrics to consider during performance appraisals. Employee engagement, attendance, and communicative skills can all be considered great metrics to track. However, when the chips are down, all anyone will care about is the quality of your work. The end result! Here are some performance review phrases on the quality of work.

  • “Produces consistently high-quality work that meets or exceeds expectations.”
  • “Pays attention to detail, catching errors or issues before they become problems.”
  • “Shows pride in work, going above and beyond to ensure excellence.”
  • “Demonstrates a strong understanding of requirements and produces work that aligns with them.”
  • “Is committed to continuous improvement, regularly seeking feedback and making adjustments.”
  • “Has a hard time consistently producing work that meets expectations and may require additional support or revisions.”
  • “Has challenges in maintaining attention to detail, which can result in errors or oversights.”
  • “Has a tendency to rush work, resulting in lower quality outcomes.”
  • “Needs to take greater ownership of their work and ensure it aligns with the required standards.”
  • “Could benefit from additional training or support to enhance skills and produce higher quality work.”

4. Dependability

Nothing is certain but death and taxes, right? Dependability is an extremely important performance review criterion. It is, however, a bit hard to pinpoint. How does one measure dependability? Is it taking initiative, showing a strong work ethic, or simply taking responsibility? Well, here are some sample performance evaluation phrases that cover all those bases!

  • “Is a reliable team member, consistently meeting commitments and delivering high-quality work.”
  • “Takes responsibility for tasks and projects, ensuring they are completed on time and to the required standard.”
  • “Brings with them a strong work ethic, putting in extra effort when required to meet team goals.”
  • “Shows initiative in taking on additional responsibilities and supporting team members.”
  • “Is committed to continuous improvement, actively seeking feedback and making adjustments.”
  • “Has trouble keeping commitments, occasionally needing more assistance or time extensions.”
  • “Has difficulty accepting responsibility for jobs or projects, occasionally blaming others for mistakes or delays.”
  • “Tends to miss deadlines or deliver work that is below the required standard.”
  • “May need to improve time management and planning skills to better meet expectations.”
  • “Could benefit from increased accountability and taking ownership of mistakes or challenges.”

5. Initiative

Taking initiative means being proactive, taking ownership of our work, and seeking out new opportunities. It’s what sets us apart and makes us stand out as top performers. Your performance review comments should definitely include performance appraisal phrases about taking initiative. So here are some performance review comments examples centered around initiative!

  • “Takes initiative to identify and address challenges or opportunities without being prompted.”
  • “Shows creativity and innovation in identifying new solutions or approaches to tasks and projects.”
  • “Has the willingness to take on new challenges and responsibilities.”
  • “Is proactive in identifying and addressing potential issues before they become problems.”
  • “Seeks out opportunities to improve processes or procedures, contributing to overall team success.”
  • “Often requires a nudge to take the reins and identify opportunities for improvement.”
  • “Encounters roadblocks when it comes to brainstorming innovative solutions and thinking outside the box.”
  • “May benefit from a confidence boost to take on new challenges and responsibilities with enthusiasm.”
  • “Could leverage growth opportunities by seeking out challenges and embracing new experiences.”
  • “Needs to take a more proactive approach to identify and tackle potential issues before they escalate.”

6. Teamwork

“Teamwork makes the dream work,” as they say. But let’s be honest, working in a team can be challenging at times. With so many different personalities and working styles, conflicts are bound to arise. However, when we work together effectively, we can achieve great things. In this section, we’ll take a look at some performance review comments related to teamwork and collaboration.

  • “Is a supportive team member, working collaboratively to achieve team goals.”
  • “Contributes positively to team dynamics, fostering a positive work environment.”
  • “Interacts effectively with team members, building strong relationships and fostering a sense of camaraderie.”
  • “Shows a willingness to help colleagues and offer support when needed.”
  • “Respects and values diverse perspectives, contributing to an inclusive and welcoming team environment.”
  • “Faces hurdles in working collaboratively with team members, at times operating in groups or generating discord.”
  • “Has some room for growth in terms of communication skills to foster stronger connections with colleagues.”
  • “Shows a tendency to prioritize individual objectives over team goals, which can hinder overall team success.”
  • “Has potential to grow by demonstrating more empathy and support towards colleagues.”
  • “Requires development in valuing diverse perspectives and creating an inclusive team environment.”

7. Leadership

Being a leader isn’t just about giving orders and bossing people around. It’s about inspiring and motivating others, setting goals, and guiding your team to success. Of course, it’s not always easy to be a great leader. It takes patience, empathy, and a willingness to learn and grow. So, why don’t we explore some performance review comments that relate to leadership skills?

  • “Demonstrates strong leadership skills, inspiring and motivating team members towards success.”
  • “Shows a commitment to developing team members, providing support and opportunities for growth.”
  • “Leads by example, modeling positive behavior and work ethic for team members to follow.”
  • “Delageates tasks clearly and effectively to team members, setting clear expectations and goals.”
  • “Effectively manages conflicts and challenges, finding solutions that benefit the team as a whole.”
  • “Has a difficult time leading effectively, sometimes causing confusion or conflict among team members.”
  • “Has issues with communicating expectations or providing clear direction to team members.”
  • “Needs to improve on supporting and developing team members, leading to a lack of motivation or engagement.”
  • “Could benefit from developing stronger conflict resolution and problem-solving skills.”
  • “Needs to work on modeling positive behavior and work ethic for team members to follow.”

8. Adaptability

No matter what line of work you’re in, it doesn’t take a sociologist to see that we are going through one of the most volatile times in human history. It feels like there is a new paradigm shift every single week! That is why adaptability is a great category of performance review comments. As a performance appraisal metric, the adaptability of an employee needs to be measured and given feedback upon! So here are some performance review comments on adaptability!

  • “Is able to adapt quickly and effectively to changing priorities or circumstances.”
  • “Shows flexibility in approach, willing to adjust plans or strategies as needed to achieve goals.”
  • “Demonstrates resilience in the face of challenges, persevering to achieve success.”
  • “Handles uncertainty and ambiguity with ease, remaining focused and productive.”
  • “Thrives in a fast-paced environment, showing energy and enthusiasm for new opportunities.”
  • “Fails to adjust to changing conditions or priorities, resulting in missed opportunities or inefficiencies.”
  • “Can definitely use some work on being more flexible and open-minded in approach.”
  • “Can become overwhelmed by uncertainty or ambiguity, leading to decreased productivity.”
  • “Needs to improve resilience and persistence in the face of challenges or setbacks.”
  • “Could benefit from developing strategies for managing stress and pressure in a fast-paced environment.”

9. Problem-Solving

When we encounter problems in the workplace, it can be easy to feel overwhelmed or unsure of how to proceed. But being able to think creatively and come up with innovative solutions is an essential skill for any employee. It can save time, increase productivity, and make us all look like superheroes. So, let’s dive into some performance review comments related to problem-solving skills.

  • “Demonstrates excellent problem-solving skills, able to analyze complex problems and develop effective solutions.”
  • “Approaches problems systematically and logically, using a variety of resources and tools to find solutions.”
  • “Is able to identify root causes of problems, rather than just treating symptoms.”
  • “Collaborates effectively with colleagues to find solutions to difficult problems.”
  • “Is innovative and creative in finding new solutions to problems.”
  • “Complex problem-solving can be a challenge, resulting in possible delays or inefficiencies.”
  • “Usually treats symptoms instead of investigating the underlying causes of problems.”
  • “Developing a more systematic and logical problem-solving approach could be beneficial.”
  • “Collaborative skills may need improvement to effectively work with colleagues in problem-solving.”
  • “Enhancing innovation and creativity in problem-solving is an area that requires attention.”

10. Creativity

Did we save the best for last? Yes, we did! The importance of creativity as a key performance metric is constantly growing. In fact, with the emergence of AI productivity software, true creativity is one skill the robots can’t seem to emulate just yet. So why shouldn’t you have a list of performance review comments focused solely on creativity? Of course, you should!

  • Displays creativity that is inspiring and has had a significant impact on the success of the project.
  • Demonstrates an ability to think outside the box and come up with innovative ideas that help the team overcome challenges and achieve goals.
  • Consistently brings fresh perspectives to the table and takes risks in order to achieve great results.
  • Applies a creative approach to problem-solving that leads to unique solutions, improving processes and saving time and resources.
  • Shows enthusiasm for exploring new ideas and experimenting with different approaches, fostering a culture of innovation within the team.
  • While technically sound, work lacks the creative flair that would make it stand out from the competition.
  • Provides solutions to problems that are often formulaic and lack originality.
  • Appears to be stuck in a rut and is not coming up with new ideas or approaches to address challenges.
  • Resists change and is unwilling to experiment with new ideas, which holds the team back.
  • Displays creativity that is limited to a particular style or medium, and does not demonstrate the ability to adapt to new situations and come up with fresh ideas.

Digitize Your Performance Reviews

Do you know what can be equally important as the performance review comments you use? How you actually conduct those performance reviews? If your team has a terrible time with performance reviews and they hate partaking in them, then your performance review comments won’t have any meaning. The best way to conduct your performance reviews today is to digitize them!

You can read our guide on digitizing your performance reviews right here!

Performance Review Software

A massive part of digitizing your performance reviews is to make use of performance review software . There are many incredible alternatives available to users worldwide and you’re never going to believe it… We made a list of the best of ’em! Just for you. Right here: Top 10 Performance Review Software of 2023 . If you’re old-fashioned, we also have something for you: Excel Performance Review Templates .

Performance review comments : Teamflect performance reviews example with questions in microsoft teams

If you are a Microsoft Teams user, then you don’t need to look any further. Teamflect is the best performance review software for Microsoft Teams. Teamflect’s complete Microsoft Teams integration allows for everything to stay in the flow of work. You can conduct entire performance review cycles, without ever having to leave Teams.

One of the biggest problems with performance management in 2023, surprise surprise, isn’t performance review comments! It is the dreadful practice of juggling multiple software at the same time. Teamflect doesn’t just let you conduct performance reviews inside Microsoft Teams.

It also gives you access to a massive performance review template gallery , filled to the brim with customizable templates and performance review comments galore!

Teamflect Image

Conducting Performance Appraisals in Microsoft Teams

Integrating your performance appraisals into the best communication and collaboration hub there is always a safe bet. That is something you just can’t achieve through analog performance review methods.

Here is how you can use Teamflect to complete an entire performance review cycle in a matter of clicks!

Step 1: Go into Teamflect’s Reviews Module

Teamflect’s interface is incredibly easy to navigate. In order to start a review cycle, all you have to do is click “New Review” once you’re in the “Reviews” module.

This module is also home to all performance reviews conducted in your organization. It functions both as a central hub and an archive for performance appraisals.

image 20 2

Step 2: Choosing Your Performance Review Template

Teamflect has an extensive library of performance review templates for you to choose from. While they are ready to be used as is, you can always customize them to fit your organization’s needs.

Some of the ways you can customize Teamflect’s review templates include:

  • Changing question types: Open-ended, Multiple Choice, Likert Scale, Rating, etc.
  • Integrating goal completion rates.
  • Integrating 360-degree feedback data.
  • Including an employee development plan .
  • Creating a custom evaluation criteria
  • Adding the 9-Box Talent Grid.

image 20 1

Step 3: Complete The Review

Once you send out your performance review template of choice, all that is left to do is for both parties to fill out their ends of the performance review. That is how easy it is to conduct an entire performance review inside Microsoft Teams with Teamflect.

image 20

Automating Review Cycles

Teamflect users have the option to create custom automation scenarios for their performance reviews. While we opted to show you the manual process of conducting performance appraisals with Teamflect, you can also queue all of the steps shown above to a custom automation scenario and let those reviews be automatically sent to reviewees at your desired intervals!

image 19

How to write performance review comments?

When you’re writing performance review comments, there are some things you should make sure your performance review comments always are. Three key things to keep in mind as you’re writing your performance review comments are:

1. Start with positive feedback: Begin your comments with positive feedback to set a constructive tone and make the employee feel valued.

2. Focus on behavior: Focus on the employee’s behavior, rather than their personality. This helps the feedback feel less personal and more actionable.

3. Avoid vague language: Avoid using vague language like “good job” or “needs improvement.” Instead, be specific about what the employee did well or what they need to work on.

What Not to Say in Performance Reviews

Performance reviews are a crucial part of employee development and growth within any organization. They provide an opportunity for constructive feedback and goal-setting.

Delivering effective performance appraisal comments can be challenging, and using the wrong words or phrases can have a detrimental impact on your team’s morale and productivity.

So in this particular section, we’ll explore what you shouldn’t say in performance reviews and provide examples to steer clear of these pitfalls.

Negative Language

Avoid using harsh or negative language in your performance appraisal comments. Phrases like “You always” or “You never” can be demotivating and unproductive. Instead, focus on specific behaviors or incidents and provide constructive feedback.

Example: Instead of saying, “You always miss deadlines,” try, “I noticed a few instances where deadlines were not met. Let’s work together to improve your time management skills.”

Comparative Statements

Refrain from making direct comparisons between employees in your performance review examples. Using phrases like “You’re not as good as [colleague]” can create unhealthy competition and resentment among team members. Instead, concentrate on individual strengths and areas for improvement.

Example: Avoid saying, “You’re not as efficient as Sarah,” and opt for, “I believe you can further improve your efficiency by implementing time-saving techniques.”

Vague Feedback

Performance review phrases that lack specificity can be frustrating for employees. Avoid vague comments like, “You need to do better” or “Your work has room for improvement.” Instead, provide clear examples and suggest actionable steps for improvement.

Example: Say, “Your recent project lacked detailed documentation. To improve, please make sure to document all processes thoroughly for future projects.”

Personal Criticisms

Keep your performance appraisal comments focused on work-related matters. Avoid making personal criticisms or judgments about an employee’s character or personality.

Example: Don’t say “You’re too introverted for this role,” and opt for, “To excel in this role, consider taking on more proactive communication and teamwork initiatives.”

Unsubstantiated Claims

Ensure that your performance review examples are based on observable and documented behaviors. Avoid making unsupported claims or accusations.

Example: Instead of stating, “You’re always late,” provide evidence like, “I’ve noticed on three occasions this month that you arrived late to our team meetings.”

Ambiguous Praise

While praise is essential, ambiguous compliments can be ineffective. Avoid phrases like “You’re doing great” without specifying what the employee is excelling at. Instead, be specific and highlight their accomplishments.

Example: Say, “Your recent project presentation was outstanding. Your attention to detail and engaging delivery truly impressed the team.”

How to make supervisor comments and recommendations?

As a supervisor, it is your job to offer supervisor comments and recommendations to your direct reports. While this may seem like a mundane task at first, effective supervisor comments and recommendations can prove to be the most valuable out of all your performance review comments.

1. What is the point of your supervisor comments and recommendations?

Many leaders fall under the false impression that since they are a supervisor, they are under the obligation to offer comments and recommendations. Unless there is a distinct purpose behind your supervisor comments and recommendations, you should abstain from handing them out. Or you will start skirting the dangerous line into the micro-management territory!

2. Do your homework first!

To make sure your comments and recommendations as a supervisor are hitting the mark, you need to do your research and do it well! Some of the best ways to gather that information include:

  • Pulse Surveys
  • 360-Degree Feedback
  • Anonymous Feedback
  • Check-in Meetings

Here is a quick video tutorial on just how you can gather 360-degree feedback inside Microsoft Teams:

3. Focus on solutions instead of problems.

Instead of just pointing out problems, provide recommendations for improvement. Better yet, turn your feedback session into a discussion where you and your direct reports brainstorm on fixing any existing problems together. Supervisor comments and recommendations are opportunities to show your team that you’re in it together.

Related Posts:

problem solving for evaluation

Written by Emre Ok

Emre is a content writer at Teamflect who aims to share fun and unique insight into the world of performance management.

race track

Best Performance Review Tips for Employees – 2024

types of employee recognition

6 Types of Employee Recognition: Best Practices and Examples! – 2024

Problem Solving Performance Review Phrases Examples

Related Evaluation Phrases

Problem Solving Sample Phrases To Write A Performance Appraisal Feedback

300 Performance Review Phrases in 20 Areas of Work

problem solving for evaluation

Performance review season is a stressful time of year. As a leader, it can be a struggle for you to properly describe a behavior exhibited by an employee, so we tried to make it easier by listing over 100 performance review phrases below. They are grouped by category to make it easy to find inspiration and not get in trouble with HR.

Two pieces of advice before you get started:

At Friday, we make it easy to have regular conversations, so you don’t have to bottle up your feedback for a review and performance appraisal that happens once a year.  See our playbooks  for success. You should not delay important conversations. We strongly recommend having regular  1-1s  and establishing a feedback loop. Don't kick the can down the road.

Second, please make sure when completing a performance review that you  outline behaviors, not personality traits . Behaviors can be changed. Also make sure you document the behavior. Ideally, you bring it up in-the-moment vs. a performance review.

With the disclaimer out of the way, here's a list of canned performance review phrases to give you some inspiration. For the love of everything, please don't copy these phrases verbatim.

You owe it to your employee to put some time into this process and consider their overall performance. If you want to learn how to effective run a performance review, you should read  High Output Management .

Before we dive in take a look at these performance review templates !

Active Listening

1. Is an active and focused listener

2. Makes sure the team feels heard

3. Creates healthy dialogue to help the best solution come forward

4. Listens with an open mind

5. Demonstrates a sincere appreciation for opposing viewpoints

6. Actively seeks feedback, even when it’s not in agreement

7. Asks insightful questions to understand the root cause

8. Empathizes with others who have opposing viewpoints

9. Follows instructions with care and attention

Needs Improvement

10. Asks questions that show he/she is not actively listening to the conversation at hand

11. Is distracted easily and doesn’t fully absorb another person’s point of view

12. Interrupts peers

13. Constantly interjects into conversations

14. Dismisses other people’s ideas who she/he doesn’t agree with

15. Stops listening when presented with an opposing viewpoint

Administration

16. Is effective when handling large amounts of data 

17. Produces high-quality work that is well documented and detailed
 

18. Is capable of working independently from day one

19. Is a strong team player
 

20. Is well organized, and can effectively implement projects from start to finish
 

21. Can make independent decisions when necessary
 

22. Demonstrates good time management skills
 

23. Can adapt his/her schedule to meet the needs of the business
 

24. Has excellent follow-through skills, and is always on time with deliverables 

25. Shows ability to be proactive in identifying tasks that need to be completed 

Areas for Improvement 

26. Does not show an aptitude for number crunching and data entry activities 

27. Has difficulty following through with tasks that require a high amount of organization and time management 

28. Seemed to have trouble getting organized this week
 

29. Demonstrates disinterest in the administrative aspects of the job 

30. Has trouble with filing, record keeping, and other administrative tasks


31. Is less than eager to complete documentation-related work
 

32. Is often late, which negatively impacts project completion timelines

Attendance & Being On-Time

33. Is punctual and consistently on-time to meetings

34. Arrives at meetings on time and is always prepared

35. Attained perfect attendance over $time_period

36. Schedules time-off according to company policy

37. Completes deadlines as promised

Areas for improvement

38. Does not meet attendance standards for punctuality

39. Has exceeded the maximum number of vacation days allotted

40. Is frequently late to work

41. Does not return communications in a timely manner

42. Is a quick study 

43. Learns from mistakes 

44. Is willing to hear constructive criticism 

45. Accepts coaching well, and does not become defensive when given negative feedback 

46. Takes responsibility for own mistakes and failures 

47. Demonstrates a willingness to improve performance in the face of negative feedback 7. Asks questions about the most effective way to approach new activities 

48. Asks for clarification when he or she is unsure how to correctly complete a task or activity 

49. Demonstrates self-awareness and willingness to improve in areas where he or she is lacking 

50. Is sensitive to the fact that there is always room for improvement, both personally and professionally 

Areas of Improvement

51. Seeks individual coaching only when an area of weakness has been identified by someone else 

52. Demonstrates lack of awareness regarding his or her own deficiencies as a team member or employee 

Communication

53. Regularly gives constructive feedback

54. Is willing to entertain others ideas

55. Asks great questions

56. Explains tough issues in a way that is clear to the rest of the team and direct reports

57. Is not afraid to say “I don’t know” when presented with a tough question.

58. Is effective at summarizing and communicating key business decisions.

59. Stands out among his/her peers for outstanding communication skills

60. Is effective at persuading and convincing peers, especially when the stakes are high

61. Is an effective listener, always willing to listen and understand peers objections

62. Makes new employees feel welcome

63. Acts as a calming force when the team is under pressure

64. Blames others for problems

65. Complains of lack of resources to adequately complete job functions

66. Fails to alert proper personnel regarding bad news

67. Regularly engages in off-putting conversations, can be territorial at times

68. Humiliates staff members on a regular basis

69. Has trouble communicating effectively in groups

70. Has other coworkers deliver bad news instead of doing it himself/herself

71. Makes others feel intimidated when working on company projects

Cooperation

72. Is easy to work with, and encourages others to work together as a team 

73. Is supportive of coworkers, even in the face of conflict or adversity 

74. Supports group decisions even when it is not his/her preferred course of action 

75. Fosters a cooperative atmosphere 

76. Is a team player who does not put personal goals ahead of the greater good of the group 

77. Is receptive to ideas from peers and willing to adapt his/her own behaviors in response

78. Frequently expresses frustration with coworkers when they are too slow, or do not complete their tasks in time for the next project phase to begin on time 

79. Frequently comes late to meetings and disrupts workflow for other workers 

80. Takes credit for the work of others, especially when it is not deserved 

81. Does not share information with colleagues if they do not ask for it directly 

82. Thinks outside the box to find the best solution to a particular problem

83. Is creative and finds ways to correlate ideas with action

84. Artfully changes when presented with new information and ideas

85. Is always willing to directly challenge the status quo in pursuit of a more effective solution

86. Contributes fresh ideas regularly

87. Encourages coworkers to be inventive

88. Contributes innovative ideas in group projects

89. Contributes suggestions regularly on how to improve company processes

90. Demonstrates disinterest in contributing creative or innovative ideas

91. Is rigid and unwilling to adjust when presented with new information

92. Fails to properly attribute coworkers who contribute innovative solutions

93. Seems unwilling to take risks, even when presented with a compelling reason

94. Fails to incentivize peers to take creative and innovative risks

95. Is reluctant to find more effective ways to do job activities

Customer Relations

96. Works effectively with clients

97. Has strong rapport with those he/she interacts with

98. Enjoys the people related aspects of the business

99. Is pleasant and projects a friendly tone over the phone

100. Has become a linchpin with clients

101. Consistently spearheads effective customer relations

102. Is empathetic towards customer issues

103. Is direct, yet helpful in dealing with customer concerns

104. Displays an effective cadence when working with clients on projects. Is effective, not overbearing

105. Artfully helps customers overcome objections

106. Can handle difficult customers with grace

107. Consistently receives substandard comments from customers

108. Appears disinterested in helping customers with their challenges

109. Does not manage customer expectations, especially in tough situations

110. Consistently passes challenging issues to others instead of tackling them head-on

111. Misses opportunities to further educate customers about other products or services.

112. Uses inappropriate language with colleagues or customers

113. Appears to become frustrated by clients who ask questions

114. Displays sarcasm when dealing with client challenges

115. Is a good delegator, even with those who don't report directly to him/her 

116. Is able to delegate tasks and responsibilities effectively to appropriate team members 3. Is able to delegate work without micromanaging 

117. Is able to delegate work while maintaining the vision of what is to be accomplished 

118. Is able to delegate work to teammates without fear of losing control 

119. Demonstrates trust in subordinates, allowing them the freedom to make decisions 

120. Designs delegation strategies which provide opportunities for staff members to grow and develop 

121. Seems unwilling, or unable, to give up control of work activities completely when delegating tasks

122. Appears hesitant about allowing subordinates the freedom needed in order to complete delegated tasks successfully 

123. Does not seem confident when delegating tasks or responsibilities; does not inspire confidence in subordinates when delegating work

Flexibility

124. Constantly identifies more efficient ways of doing business


125. Is accepting of constructive criticism


126. Is a well-versed team player capable of handling a variety of assignments


127. Is calm under pressure


128. Is a calming force, especially with [his/her] peer group


129. Shows initiative, and is flexible when approaching new tasks

130. Does not excel at activities which require a high degree of flexibility

131. Tends to resist activities where the path is unknown

132. Appears uninterested in new duties

133. Becomes uptight when the plan changes

Goal-Setting

134. Is effective at goal-setting and challenging oneself

135. Clearly communicates goals and objections to coworkers

136. Is constantly striving to be the best he/she possibly can be

137. Sets concrete and measurable goals

138. Sets aggressive targets to meet business objectives

139. Creates clearly defined goals that align with the company’s mission

140. Proactively shares progress on goals

141. Is inconsistent in defining goals and objectives

142. Struggles to set goals that align with company objectives

143. Sets performance goals that are out of touch with reality

144. Struggles to communicate when deadlines will be missed

145. Leaves peers struggling to understand the status of a project

146. Refuses to delegate to others, attempts to do all the work by himself/herself

147. Is unwilling to claim responsibility for missed goals

148. Is easily distracted and disinterested in focusing to achieve performance goals

149. Finds creative ways to solve problems and improve processes 

150. Is an idea generator, and finds innovative ways to accomplish tasks 

151. Contributes new ideas to team project discussions regularly 

152. Contributes new ideas at staff meetings  

153. Contributes well-thought out suggestions for company improvements 

154. Evolves process improvement ideas in staff meetings 

155. Thinks outside the box when presented with a new challenge

Areas of Improvement 

156. Tends to react negatively when presented with a new problem or challenge which requires innovative thinking 

157. Fails to offer new solutions even when presented with a compelling reason to do so 10. Has difficulty coming up with creative ideas, even when prompted by supervisor 

158. Fails to recognize that all employees should contribute innovative ideas for improvement of company procedures and processes

Create improvement plans with 30-60-90 day templates !

Interpersonal skills

159. Is a good communicator with [his/her] peers, requiring little or no direction in how to communicate 

160.Is a strong team player who thrives on teamwork 

161. Has the ability to diffuse conflict among peers 

162. Has the ability to resolve emotional situations quickly and efficiently 

163. Is able to hear and accurately interpret verbal and nonverbal cues from peers

164. Has the ability to diplomatically handle difficult situations with peers 

165. Has the ability to ensure that [his/her] peers are satisfied with a decision made by a superior

166. Is able to effectively communicate across cultures, geographical regions, etc.

167. Utilizes peer feedback to make improvements in [his/her] interpersonal skills 

168. Is an active listener that is sensitive to both verbal and non-verbal cues from peers 22. Has the ability to approach every situation with confidence 

169. Fails to appropriately respond when receiving negative feedback from coworkers 24. Fails to ask for clarification if [he/she] does not understand what a coworker is trying to tell [him/her] 

170. Does not use constructive criticism from peers in an effort to improve performance at work or in school

171. Does not take initiative or act on positive suggestions given by coworkers68. Uses abrasive language when interacting with peers

172. Displays defensiveness when receiving constructive criticism

173. Appears uncomfortable when asked questions during group discussion

174. Shows sound judgment when evaluating multiple opportunities

175. Comes to reasonable conclusions based on information presented

176. Is fact-based, and refuses to accept emotional arguments when evaluating a decision

177. Remains calm, especially under stress

178. Balances swift decision-making, with the ability to analyze the many angles to a story

179. Is confident and persuasive when making big decisions

180. Consistently understands the core issues at play, enabling him/her to solve problems at a remarkable pace

181. Makes confident decisions when presented with facts and data.

182. Effectively prioritizes urgent matters with those that can wait

183. Effectively outlines the best case (and worst case) scenarios to aid decision-making

184. Makes hasty decisions without first collecting facts & data to inform the decision-making process

185. Consistently displays analysis paralysis when making a decision

186. Makes big decisions without approval from respective parties

187. Is unable to keep confidential information private

188. Approaches decisions with a one-track mindset. Has a “my way or the highway” view of others ideas.

Leadership Ability

189. Is a servant-leader, always willing to help his/her team

190. Gives structure, feedback, and direction to his/her team

191. Consistently recognizes his/her team for a job well done

192. Actively listens to his/her team

193. Creates a culture of dialogue

194. Recognizes staff for a job well done

195. Provides just enough conflict to find the best outcome

196. Balances the strategy of the organization with tactical day-to-day tasks

197. Demonstrates a high bar for ethical behavior

198. Tends to overanalyze problems when a prompt decision is required

199. Fails to plan for the future

200. Sends mixed signals to the team regarding goals and day-to-day activities

201. Rarely gives recognition to his/her team

202. Sets an unreasonably high expectation for his/her team

203. Fails to keep confidential information secret

204. Plays favorites and does not treat each member of the team equally

205. Shows interest in learning new skills and expanding knowledge base 

206. Is willing to take risks to gain new skills and knowledge 

207. Consistently seeks out professional development opportunities 

208. Demonstrates a desire to learn from peers rather than simply teaching them 

209. Is able to learn from past mistakes 

210. Consistently seeks out new experiences 

Needs Improvement 

211. Rarely uses past experience as a guide for decision making in new situations 

212. Does not appear willing to take risks to gain new skills and knowledge 

213. Fails to understand the value of taking risks to gain new skills and knowledge 

214. Has difficulty accepting criticism 

215. Does not like to be told what to do, when, or how to do it

Management skills

216. Is self-motivated

217. Manages time well

218. Sets attainable goals and objectives

219. Is able to effectively prioritize amongst competing demands

220. Is a good negotiator, able to get people to agree to new ideas or actions

221. Communicates clearly and delivers messages in concise, effective ways

222. Recognizes the contributions of peers and subordinates

223. Assigns challenging but appropriate tasks to staff, and provides adequate resources to ensure successful completion of assignments

224. Has difficulty being firm when needed with staff who are underperforming or inefficient 

225. Fails to reward subordinates for hard work or for delivering results on time 

226. Perceives subordinates as too personal and not business-like in their approach to work relationships 

227. Views employees as a cost center rather than an asset for the company's future growth and profitability

Motivation & Drive

228. Constantly pursues learning opportunities

229. Consistently takes on additional responsibility for the team

230. Successfully finds more effective ways to perform a specific task

231. Is constantly looking for new ways to help the team

232. Is always willing to jump in and learn something new

233. Requires little direction when given a new responsibility

234. Is not afraid to take calculated risks

235. Is not afraid to make periodic mistakes

236. Is unwilling to assume responsibilities outside of his/her job description

237. Frequently sows seeds of doubt with the rest of the team

238. Resists opportunities to train and learn new things

239. Contributes few suggestions to projects with ambiguity

240. Can be overzealous, stepping on others’ toes

241. Seems unwilling to learn new things

242. Struggles to do tasks without assistance from peers

243. Is a careful planner, and always considers the end result of [his/her] actions

244. Creates detailed plans to ensure [his/her] work is done efficiently and effectively 

245. Does not leave any loose ends when completing assigned tasks

246. Is able to accurately predict the outcome of [his/her] actions 

247. Is an effective time manager 

248. Sets realistic deadlines for [his/her] work, and does not procrastinate 

249. Does not waste time, and completes [his/her] work efficiently 

250. Is organized, and keeps notes or files with information that may be needed in the future

251. Has difficulty deviating from a previously established plan when new information is revealed 

252. Does not demonstrate much concern for the long-term effects of [his/her] decisions 11. Is slow to start working on an assignment until all details are ironed out 

253. Has difficulty making quick decisions under pressure 

254. Waits until last minute to begin work on an assignment, and becomes stressed during crunch time 

255. Appears disorganized, even though [he/she] is very organized in reality 

256. Appears to lack organization skills in front of peers and upper management 

257. Is unable to prioritize tasks appropriately based on urgency and importance 80. Is constantly late with assignments and projects 

Problem-Solving Skills  

258. Sees problems as challenges to be overcome

259. Brings ideas to the table when discussing problem-solving 

260. Is able to pressure test his/her own ideas in a calm but assertive way 

261. Is able to persuade others that [his/her] solutions are the best course of action 

262. Is able to persuade others with [his/her] ideas 

263. Sets up a system for tracking problems and their solutions 

264. Uses a variety of techniques to solve problems, and is willing to try new approaches 8. Puts in extra time to ensure that problems are solved completely 

265. Is able to explain the rationale behind [his/her] solution to a problem 

266. Demonstrates effective leadership skills when solving group problems 

267. Demonstrates disinterest in pressure testing ideas with others 

268. Does not set up a system for tracking problems and their solutions 

269. Fails to take action when a problem is raised 

270. Fails to explain the rationale behind [his/her] solution to a problem 

271. Does not optimize use of time when working on difficult problems 

272. Does not solve problems in an efficient manner 

273. Is not adept at persuading others to buy into [his/her] ideas 

274. Fails to persuade others that [his/her] solutions are the best course of action 

275. Is unable to pressure test his/her own ideas in a calm but assertive way

Time Management

276. Handles multiple assignments and projects well with limited supervision 

277. Is able to manage numerous assignments with great speed and accuracy 

278. Can effectively prioritize work so that the most critical assignments receive attention first 

279. Manages his/her time in a highly effective manner 

280. Is able to complete assignments even with extended deadlines 

281. Displays a strong ability to manage multiple assignments and projects simultaneously 

282. Does not meet deadlines, even when given ample time 

283. Has difficulty balancing multiple assignments and projects 

284. Does not manage his/her time effectively 

285. Fails to effectively prioritize work so that the most critical assignments receive attention first 

286. Displays a tendency to allow personal issues affect work performance 

Understanding of Job Responsibilities

287. Has a strong understanding of job responsibilities

288. Regularly alerts management of key developments in his/her job function

289. Has deep knowledge that surpasses job expectations

290. Regularly contributes and works with other departments

291. Crafts an extensive network of peers to tackle tough issues

292. Shares knowledge with peers

293. Is constantly sharing industry trends and best practices to create outsized outcomes

294. Is adept in all areas of job responsibility

295. Has little understanding of the competitive landscape

296. Has difficulty locating necessary information to complete job responsibilities

297. Produces many unnecessary errors

298. Produces substandard work

299. Fails to demonstrate a strategic mindset

300. Does not demonstrate mastery of basic concepts in the role

That’s all for now. We’ll continue to update this list of phrases in the future. Again, we strongly recommend offering continuous feedback with your team vs. waiting for an annual performance review. Also, offer an action plan to help all parties navigate difficult situations.

problem solving for evaluation

ClickUp - Your new home for work

Stay connected & aligned with a cloud hq..

right-icon

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus convallis sem tellus, vitae egestas felis vestibule ut.

Error message details.

Reuse Permissions

Request permission to republish or redistribute SHRM content and materials.

The Performance Review Problem

As the arcane annual assessment earns a failing grade, employers struggle to create a better system to measure and motivate their workers.

​After an annual review that lasted about 10 minutes, a New Jersey-based account coordinator knew it was time to leave the public relations agency where he had worked for almost a year. 

The 25-year-old, who requested anonymity, asked for the meeting because his boss had not mentioned any formal assessment process, nor had his manager ever critiqued his work. The coordinator says he sat with a trio of senior executives who did not ask him any questions beyond how he would rate himself. He says they ignored his requests for guidance on how to advance at the agency. 

Screen Shot 2023-03-15 at 85749 AM.png

This example also illustrates one of the common failures in performance management: limiting reviews to once or twice a year without having any other meaningful career discussions in between. Nearly half (49 percent) of companies give annual or semiannual reviews, according to a study of 1,000 full-time U.S. employees released late last year by software company Workhuman. 

The only situation that is worse than doing one review per year is doing none at all, experts say. The good news is that only 7 percent of companies are keeping employees in the dark about their performance, and 28 percent of organizations are conducting assessments quarterly, the Workhuman study found.  

A Pervasive Problem

Reviews generally do not work.

That doesn’t mean that more-frequent formal meetings or casual sit-downs between supervisors and their direct reports are solving the performance review quandary, either. Only about 1 in 4 companies in North America (26 percent) said their performance management systems were effective, according to a survey of 837 companies conducted last fall by consulting firm WTW. And only one-third of the organizations said employees felt their efforts were evaluated fairly. 

Meanwhile, a Gallup survey conducted last year found that 95 percent of managers are dissatisfied with their organization’s review system.

The problem is not new, though it is taking on greater importance, experts say. Millennials and members of Generation Z crave feedback and are focused on career development. Meanwhile, the tight labor market has companies searching for ways to keep high-performing employees in the fold. Fewer than 20 percent of employees feel inspired by their reviews, and disengaged employees cost U.S. companies a collective $1.6 trillion a year, according to Gallup.

Lesli Jennings, a senior director at WTW, says part of the issue is that reviews are now so much more than a discussion of past performance. They include conversations about career development, employee experience and compensation. 

“The performance management design itself is not evolving as quickly as the objectives and the purpose that we have set out for what we want it to do,” Jennings says. 

Screen Shot 2023-03-15 at 84340 AM.png

Poor Review Practices

Some argue that means it’s time to completely scrap annual reviews and stop using scales composed of numbers or adjectives to rate employees. 

“Every single human alive today is a horribly unreliable rater of other human beings,” says Marcus Buckingham, head of people and performance research at the Roseland, N.J.-based ADP Research Institute. He says people bring their own backgrounds and personalities to bear in the reviews in what is called the “idiosyncratic rating effect.” He says the ratings managers bestow on others are more a reflection of themselves than of those they’re reviewing.

Buckingham adds that very few positions have quantifiable outcomes that can be considered a measure of competence, talent or success. It’s possible to tally a salesperson’s results or test someone’s knowledge of a computer program, he says, but he’s baffled by attempts to measure attributes such as “leadership potential.”

“I’m going to rate you on a theoretical construct like ‘strategic thinking’? Everybody knows that’s rubbish,” Buckingham says. He adds that performance reviews that offer rankings give “data that’s just bad” and insists that companies rely on data analytics because they don’t trust their managers’ judgment. But instead of working on improving their managers’ skills, he says, they put data systems in place. 

“Because we don’t educate our managers on how to have some of these conversations, we’ve decided that the solution is to give them really bad ratings systems or really bad categorization systems,” Buckingham says. 

R eviewing the Data

A mong North American employers:

  • More than 9 in 10 (93 percent) cited driving organizational performance as a key objective for performance management, yet less than half (44 percent) said their performance management program is ­meeting that objective.
  • Nearly 3 in 4 (72 percent) said ­supporting the career development of their employees is a primary objective, but only 31 percent said their performance management program was meeting that objective.
  • Less than half (49 percent) agreed that managers at their organization are ­effective at assessing the performance of their direct reports. 
  • Only 1 in 3 indicated that employees feel their performance is evaluated fairly. 
  • Just 1 in 6 (16 percent) reported having altered their performance management approach to align with remote and hybrid work models, which are rapidly becoming more prevalent.

Source: WTW 2022 Performance Reset Survey of 837 organizations worldwide, including 150 North American employers.

Data Lovers

Ratings aren’t likely to disappear anytime soon, however. “Data-driven” has become a rallying cry for companies as they seek to operate more efficiently. Organizations are trying to measure everything from sales to productivity, though such efforts can cause turmoil and hurt some individuals’ careers.

A June 2022 study of nearly 30,000 workers at an unnamed North American retail chain found that women were more likely to receive higher overall ratings than men, though women were ranked lower on “potential.” 

In that study, women were 12 percent more likely to be given the lowest rating for potential, as well as 15 percent and 28 percent less likely to receive the middle and highest potential ratings, respectively, according to the professors who conducted the study, Alan Benson of the University of Minnesota, Danielle Li of MIT and Kelly Shue of Yale. The authors also said women were 14 percent less likely to get promoted than men. “Because potential is not directly observed,” they noted, “these assessments can be highly subjective, leaving room for bias.” 

Screen Shot 2023-03-15 at 85749 AM.png

Birmingham left abruptly one afternoon and did not go in to work the next day, which he says Blizzard interpreted as his resignation. Blizzard did not respond to requests for comment.

Stack ranking became popular in the 1980s after it was embraced by General Electric. Its adoption has waned, though several tech companies continue to use it. Google and Twitter relied on stack ranking to decide who to let go in their recent rounds of layoffs, according to published reports.

Birmingham says that the system can cause anxiety and competition, which can kill team cohesion, and that arbitrary lower ratings adversely affect compensation and promotion potential. These systems can also suggest that a manager is ineffective, he says. “It implies that as managers, we basically have not done our job to hire them and train them appropriately or terminate them if they really aren’t working out.”

Birmingham says he is not opposed to ranking systems but doesn’t think they’re necessary. “I feel like the conversation about how to improve your career, what the expectations are for your job and what it will take to get to the next level are all things you can do without a rating,” he says.

Measurements Matter

Grant Pruitt, president and co-founder of Whitebox Real Estate, does not give any type of rating in his performance reviews, though he believes in using data to track his employees’ performance. “What isn’t measured can’t be managed,” says Pruitt, whose company has about 20 employees in several offices across Texas. 

At the beginning of the year, Whitebox employees set goals with their managers. Discussions are held about what benchmarks are reasonable, and these targets can be changed if there is a meaningful shift in business conditions. Team leaders hold weekly department meetings with their direct reports to discuss what’s happening and track progress. Managers hold quarterly private reviews with individuals to dig deeper into whether they’re meeting their goals and if not, why.

“Was it an achievable goal? Realistic? If it was, then what do we need to do to make sure we don’t miss it the next time?” Pruitt says. Whitebox switched to quarterly reviews about four years ago to address problems earlier and avoid having issues fester, Pruitt adds.

It’s easier to set goals for people in sales than for those in other departments, Pruitt concedes. However, he adds that executives need to brainstorm about targets they can use for other roles. For example, administrative employees can be rated on how quickly and efficiently they handle requests.

Pruitt maintains that the goal system makes it easier to respond when an employee disagrees with their manager about their performance review because there are quantitative measures to examine. The data also helps eliminate any unconscious bias a manager may have and helps ensure that a leader isn’t just giving an employee a good rating because they work out at the same gym or their children go to school together.

“I think that’s really where the numbers and the data are important,” Pruitt says. “The data doesn’t know whose kids play on the same sports team.”

Whitebox employees are also judged on how well they embrace the company’s core values, such as integrity, tenacity and coachability. Some of those values may require more-subjective judgments that can be more important than hitting quantifiable goals. 

Pruitt admits that there were occasions when he looked the other way with a few individuals who were “hitting it out of the park,” even though he believed they lacked integrity. But eventually, he had to let them go and the company lost money.

“They really came back to bite me,” Pruitt says.

Screen Shot 2023-03-15 at 84352 AM.png

Grades Are Good

Diane Dooley, CHRO of Iselin, N.J.-based World Insurance Associates LLC, also believes establishing quantitative methods to gauge employees’ performance is essential. “We are living in a world of data analytics,” she says. The broker’s roughly 2,000 employees are rated on a scale of 1 to 5.

World Insurance has taken numerous steps to remove bias from reviews. For example, last year the company conducted unconscious-bias training to help managers separate personal feelings from performance reviews. And all people managers convene to go over the reviews they’ve conducted. Dooley says that process gives everyone a chance to discuss why an employee was given a certain rank and to question some decisions. “We want to make sure we’re using the same standards,” she explains.

Currently, World Insurance conducts reviews only once a year because it has been on an acquisition binge and there hasn’t been time to institute a more frequent schedule. That will change eventually, says Dooley, who adds that she wants to introduce department grids that show how an employee’s rank compares to others’ on the team. 

“It’s just a tool that helps the department or the division understand where their people are and how we can help them collectively,” says Dooley, who has used the system at other companies. 

Dooley says she isn’t worried about World Insurance holding reviews only annually, because good managers regularly check in with their employees regardless of how frequently reviews are mandated.

Such conversations can easily fall through the cracks, however. “Managers want to manage the employees, but they get so caught up in the company’s KPIs [key performance indicators] and making sure that they’re doing everything that they need to do,” says Jennifer Currence, SHRM-SCP, CEO of WithIn Leadership, a leadership development and coaching firm in Tampa, Fla. “It’s hard to set aside the time.” 

WTW’s Jennings adds that managers sometimes avoid initiating conversations with employees who are not performing well. Such discussions are often difficult, and managers may not feel equipped to conduct them. 

“Having to address underperformers is hard work,” Jennings says. 

Additionally, experts say, coaching managers to engage in such sensitive discourse can be expensive and time-consuming.

Improve Your Performance Reviews

H ere’s how to make the review process more ­palatable for both managers and their direct reports:

  • Don’t limit conversations to once or twice per year. Every team is different, so leaders should decide what schedule is most appropriate for their departments. However, it’s important to deal with any problems as they arise; don’t let them fester.
  • Set performance goals and expectations at the beginning of the year so employees understand their responsibilities. This helps lend objectivity to the process by introducing measurable targets. However, the goals should be adjusted if there are major changes to the business or an employee’s circumstances. 
  • Explain how each employee’s position, as well as each department, fits into the company’s overall ­strategy. This will help employees understand why their job matters and why it’s important.
  • Simplify the process. There’s no need for a ­double-digit number of steps or numerous
  • questions that require long-winded answers. 
  • Consider a 360-degree approach. Input from employees’ colleagues or from other managers can help give a fuller picture of employees’ capabilities and contributions.
  • Eliminate proximity bias. You may not see some employees as often as others, especially if they work remotely, but that doesn’t mean they’re not working hard. 
  • End recency bias, which is basing a review on an employee’s most recent performance while ignoring earlier efforts. Don’t let recent mistakes overshadow the employee’s other impressive accomplishments.
  • Solicit feedback from employees. Reviews should be a two-way conversation, not a lecture.
  • Train managers to give advice calmly and helpfully. This is especially important when leaders must call out an employee’s subpar performance. 
  • Don’t discuss compensation during reviews. Employees are likely to be so focused on learning about a raise or bonus that they won’t pay much attention to anything else.

Increase Conversations

Finding the right formula for performance reviews is tricky. The company’s size, values, industry and age all play a role. Currence says businesses need to think about the frequency and purpose of these meetings. Some managers may have weekly discussions with their direct reports, but the conversations might center on status updates as opposed to performance. 

“We need to have more regular conversations,” Currence says. “There has to be a happy balance.”

San Jose, Calif.-based software maker Adobe Inc. was a pioneer when it eliminated annual reviews in 2012 after employees said assessments that look backward weren’t useful and managers lamented how time-consuming they were. Instead, Adobe introduced quarterly check-ins and did away with its numerical ratings system, even though the company is “data-driven,” according to Arden Madsen, senior director of talent management.

Screen Shot 2023-03-15 at 85749 AM.png

Adobe’s system has changed over the years as the company grew from about 11,000 employees in 2012 to around 28,000 today. In the beginning, employees were not asked a universal set of questions and the information gathered was not stored in a central place accessible to all. In 2020, Adobe instituted three or four questions that must be asked at each quarterly meeting, one of which is whether the employee has feedback for the manager. Other topics covered depend on the employee, their role and their goals.

Madsen says asking consistent questions and making reviews easily accessible are important, as internal mobility within the company has grown. 

Adobe, like many businesses, separates conversations about performance from discussions about raises and bonuses, even though they’re intertwined. 

“Money is so emotionally charged,” says WithIn Leadership’s Currence. “When we tie performance review conversations with money, we as human beings do not hear anything about performance. We only focus on the money.”    

Theresa Agovino is the workplace editor for SHRM.

Illustrations by Neil Jamieson.

Related Articles

problem solving for evaluation

Rising Demand for Workforce AI Skills Leads to Calls for Upskilling

As artificial intelligence technology continues to develop, the demand for workers with the ability to work alongside and manage AI systems will increase. This means that workers who are not able to adapt and learn these new skills will be left behind in the job market.

A vast majority of U.S. professionals  think students should be prepared to use AI upon entering the workforce.

Employers Want New Grads with AI Experience, Knowledge

A vast majority of U.S. professionals say students entering the workforce should have experience using AI and be prepared to use it in the workplace, and they expect higher education to play a critical role in that preparation.

HR Daily Newsletter

New, trends and analysis, as well as breaking news alerts, to help HR professionals do their jobs better each business day.

Success title

Success caption

Status.net

142 Employee Performance Evaluation Examples (Initiative, Flexibility, Proactiveness, Goal-Setting)

By Status.net Editorial Team on June 18, 2022 — 12 minutes to read

  • Appraisal Comments for Initiative, Proactiveness, Creativity Part 1
  • Performance Evaluation Phrases for Goal-Setting Part 2
  • Performance Evaluation Examples for Flexibility Part 3
  • Additional Resources Part 4

Employee performance evaluation is a process used to assess an employee’s job performance and to make recommendations for improvement. Evaluation results should be used to help the employee and/or supervisor improve their working relationships. There are several different types of employee evaluations, but all share the common goal of enhancing the individual’s effectiveness and productivity.

The evaluation process typically begins with an evaluation plan, which sets forth objectives, methods, and criteria for evaluating employee performance. After reviewing the results of an employee’s past evaluations, a manager should develop a reasonable expectation for future performance based on these results. This expectation should be documented in an annual review or other appropriate document. Evaluation criteria can vary depending on the position and evaluation results can be used to identify any areas in which training or development may be necessary to create an action plan.

Employee self assessment is an important part of the annual performance review process. It allows employees to reflect on their accomplishments throughout the year and gives them the opportunity to provide self evaluation of their performance. Therefore, these performance evaluation phrases and examples can also be used by employees to help guide employees in creating meaningful self assessments that accurately reflect their work and contributions to the team.

In this article you will find performance review examples for the following skills:

  • Initiative, Proactiveness, Creativity Initiative is one of the most important skills a person can have in the workplace. It is an essential part of being able to take initiative, be creative, and problem solve. These are all qualities that are essential for success in any position or field. When employees have initiative, they are more likely to be proactive and take action on tasks or projects. This leads to better productivity and overall higher levels of satisfaction in the workplace.
  • Goal-Setting Setting goals is one of the most important skills an individual can have. Good goal-setting skills help individuals stay focused and on track, manage their time more effectively, and achieve their goals. When individuals know what they want and set manageable short-term and long-term goals, they are more likely to succeed. Setting goals helps individuals stay motivated. When an individual has a clear vision of what they want to achieve, it can be extremely motivational and help them push through difficult times. Good goal-setting skills predict employee satisfaction and retention.
  • Flexibility Flexibility is an important skill in the workplace because it allows employees to be adaptable and responsive to changes in their work environment. When employees are able to shift their focus and work effectively within a changing environment, they are more likely to be successful.

Each section below contains positive (“Meets or Exceeds Expectations”) and negative (“Below Expectations”) examples for the skills listed above. If you are looking for more performance review examples and phrases for different skills, check our main article here (click to open 2000+ Performance Review Phrases: The Complete List)

Part 1 Initiative, Proactiveness, Creativity

Meets or exceeds expectations.

✓ He doesn’t wait for instructions. He shows the initiative to find new tasks himself.

✓ He requires minimal supervision. He shows initiative on his own.

✓ He s a goal-oriented person. He sets his own priorities to accomplish his job.

✓ He is a very creative person. He is skilled in finding the best ways to get a job done.

✓ He has a high sense of responsibility to his job. He tries to perfect his performance without prompting.

✓ He never feels satisfied with his performance. He always seeks new ways to improve himself.

✓ He is supportive of new ideas, goals and working methods no matter where they come from.

✓ He is always ready to take over new tasks whenever needed.

✓ Has excellent communication skills. He gets the job done through the best use of people.

✓ He is flexible and has the ability to adjust to any situation. He shows willingness to do whatever is necessary to get the task done.

✓ He always seeks ways to enhance his abilities and better himself.

✓ He always takes responsibility for his team and its performance.

✓ He always looks for new challenges and makes the work environment better.

✓ He always takes initiative in overcoming obstacles and finding a resolution that meets everyone’s needs.

✓ He fulfils his duty is by finding new challenges for himself.

✓ He does not need guidance. He always carries out his assignments without waiting to be told.

✓ He is a responsible staff member. He always performs his assignments through initiative without supervision.

✓ He explores new opportunities without being pushed to do so.

✓ He requires minimum supervision.

✓ He is always pursuing ways to further development or better himself.

✓ He always takes responsibility for his area and their actions.

✓ He is always challenging the way it has always been done and seeks to improve the environment.

✓ He champions new ideas, objectives or tools.

✓ He seeks and takes on any new opportunity that might present itself.

✓ He often thinks that his performance is not as good as everybody says. He always wants to try to improve his performance as much as possible.

✓ He is considered as the best person in the group because of his innovative ideas, critical goals and effective working methods.

✓ He never minds taking on new tasks. He always takes on even the most difficult tasks to develop himself.

More performance evaluation examples for creativity:  242 Performance Appraisal Examples for Creativity, Accountability See also: 169 Performance Review Feedback Phrases for Leadership Skills and Management Style 

Below Expectations

✗ His experience and knowledge doesn’t reflect that listed in his application.

✗ He must be closely supervised if he is to his work.

✗ He is lethargic and lacks the desire or volition to learn new skills or develop his qualifications.

✗ It seems too difficult for him to do his job on his own.

✗ He won’t improve if he continues to neglect the opportunities in front of him.

✗ He does not seem to be adaptable. He is unable to perform consistently under pressure or thrive on constant change or challenge.

✗ He typically thinks inside the box and is afraid to risk doing anything in a new way.

✗ He is indecisive. He is unable to make quick decisions, take action or commit himself to a project’s completion.

✗ He does not seem to be an ideas person. He is not ready to generate or recognize new solutions when performing a task.

✗ He has poor abilities to establish priorities and courses of action for himself. He lacks the skills in planning and following up to achieve results.

✗ He does not use his experience and knowledge to its full potential.

✗ He needs close supervision when he is performing his assignments.

✗ He does not want to pick up any new techniques or skills.

✗ He finds it difficult to perform his duties without assistance or supervision.

✗ He lacks the excellence and skills detailed in his application.

✗ He needs constant guidance in order to accomplish his assignments.

✗ He does not actively want to learn new skills or techniques as or to improve his qualification.

✗ He doesn’t apply himself as much as he could given his experience and knowledge.

✗ He requires constant supervision to get his work completed.

✗ He doesn’t appear to want to learn any new techniques or skills.

✗ He has trouble doing his tasks without help or supervision.

✗ He doesn’t seek out opportunities to learn and grow within his role.

✗ He fails to think out of the box and prefers to do things the way they have always been done.

✗ He always has issues when he has to deal with tasks alone.

✗ He often neglects unexpected opportunities and loses the chance to improve himself.

✗ He often works in an unprofessional manner. He never risks doing anything innovatively.

Related: Initiative: Performance Review Examples (1 – 5)

Part 2 Goal-Setting

✓One of his strengths is his ability to design achievable goals. He ensures those goals are all met on time.

✓He is effective at goal-setting and challenging himself.

✓He clearly communicates goals and objectives to coworkers.

✓ He takes responsibility for the performance of his staff members. He ensures the achievement of the goals as planned.

✓ He defines clear goals and expects the right performance from his group.

✓ He knows how to keep his staff focused on a plan. He assigns suitable duties to each of them.

✓ He knows to assign suitable duties to each staff member. He urges his staff to give him the results he expects.

✓ He gives frequent feedback to his staff members. He coaches them to perform as required.

✓ He constantly strives to be the best he possibly can be.

✓ He sets concrete and measurable goals.

✓ He sets aggressive targets to meet business objectives.

✓ He creates clearly defined goals aligned with the company’s mission.

✓ He proactively shares progress towards goals.

✓ He usually shares his knowledge with his staff to help them perform their duties better.

✓ He takes responsibility for his team’s work and goals.

✓ He assigns the right duty to the right staff and instructs them how to perform their assignments well.

✓ He always gives each staff member the right assignment. His staff reward his expectations in their performances.

✓ He ensures his staff understand their job responsibilities. He holds his staff accountable for their responsibilities.

✓ He clearly communicates objectives, and what is expected from them to his team members.

✓ He consistently shares feedback with his staff regarding their progress.

✓ He holds himself accountable for his team’s performance objectives and goals.

✓ He sets clear and measurable performance expectations.

✓ When working in a team, he always pays attention to the goals set. He monitors his staff’s achievements.

✓ He focuses on setting clear and achievable goals. He judges the right performance level from his group when doing their tasks.

✓ He has a strong vision for the future, both personally, and for the company.

✓ He develops actionable goals and plans how to meet them.

✓ He implements plans swiftly and effectively. He adjusts plans when something is not working.

✓ He works to promote the company’s mission and vision.

✓ He contributes to the company’s larger goals.

✓ He needs to set goals that are more challenging.

✓ He constantly pursues opportunities for growth and learning.

✗He struggles to set goals that align with company objectives.

✗ He is ineffective at pursuing his goals.

✗ He is easily distracted at work. He fails to focus on his goals, resulting in failure.

✗ He devolves responsibility for deadlines and objectives to employees instead of accepting them as supervisor.

✗ He is ineffective at setting achievable goals.

✗ He is inconsistent in defining goals and objectives.

✗ He assigns tasks to his employees without providing any information or feedback to keep them on the track.

✗ He sets performance goals that are out of touch with reality.

✗ He struggles to communicate when deadlines will be missed.

✗ He leaves peers struggling to understand the status of a project.

✗ He refuses to delegate to others. He attempts to do all the work by himself.

✗ He is unwilling to accept responsibility for missed goals.

✗ He is easily distracted. He is uninterested in focusing on achieving performance goals.

✗ He does not assign his staff effectively. His ineffective assignments mean he cannot reach assigned goals.

✗ He does not achieve goals or objectives because he does not focus on his performance.

✗ He will shift responsibility onto others for unaccomplished deadlines or goals.

✗ He sometimes does not reach the set goals.

✗ He often assigns his members duties without giving them information or feedback. The lack of information means team members do not know what they should do.

✗ He rarely achieves goals due to his inability to assign the right duties to the right person.

✗ He doesn’t always communicate the right information to his staff to ensure they are successful with their tasks.

✗ He fails to achieve the goals because he doesn’t delegate to his staff effectively.

✗ He gets distracted and doesn’t reach his goals or objectives.

✗ He will blame others for missed deadlines and objectives.

✗ He sets goals that sometimes are not achievable.

✗ As a supervisor but he doesn’t accept responsibility is shared with his staff. He believes his employees should accept all responsibility for deadlines, objectives and results.

✗ His goal-setting ability is not good. He sometimes asks his staff to complete unachievable goals.

✗ He should strive to aim a little higher when setting goals.

✗ He would benefit from reigning in goals and vision to something more achievable.

✗ He needs to set goals that more accurately match his talent level.

Related: Goal Setting: Performance Review Examples (1 – 5)

Flexibility

✓ He is ready to improve and develop necessary skills to make his job more effective. He participates in trainings and other corporate events.

✓ He never gets stressed in unexpected situations. He never complains about innovations introduced at the workplace.

✓ He is ready to make a new and carefully considered decision if the situation has changed and the previous actions have become inappropriate.

✓ He is ready to work extra hours if urgent and essential issues must be solved by the end of the day.

✓ He is always ready for business trips when necessary for his job and for improving relationships with clients.

✓ He is strong and confident but at the same time open-minded. He is always ready to consider proposals from colleagues.

✓ He is always stays aware of market changes to be able to react immediately. This awareness helps the company develop and flourish.

✓ He always takes opposite and conflicting views into account to develop proper compromise solutions.

✓ He tries to find an individual approach to each person, colleague and customer. He optimizes his work and get benefits for the company.

✓ He constantly identifies more efficient ways of doing business.

✓ He is a flexible manager and always tries to understand and respect his employees’ situations. He creates a positive working environment.

✓ He readily accepts constructive criticism.

✓ He is a well-versed team player. He is capable of handling a variety of assignments.

✓ He is calm under pressure.

✓ He is a calming influence, especially within his peer group.

✓ He shows initiative, and is flexible when approaching new tasks.

✗He does not excel at activities which require a high degree of flexibility.

✗ He cannot refuse his colleagues’ requests. He excessively takes on extra work and additional problems.

✗ He always adapts to production changes by trying to please his manager. Standing on his own would be better.

✗ He fails to focus on his main task because he tries to gain more and more additional skills at his work.

✗ His employees abuse his willingness to allow them to work from home in case of necessity.

✗ He tries to perform several tasks simultaneously to finish work faster instead of setting the right priorities.

✗ He always accepts too many tasks and cannot cope with the huge amount of work in result.

✗ He can change his point of view without analyzing or defending it. He should be more certain about his proposals.

✗ Due to his high levels of responsibility, he cannot afford to relax and enjoy some time out of work.

✗ He tends to resist activities where the path is unknown.

✗ He appears uninterested in new duties.

✗ He becomes uptight when the plan changes.

Related: Flexibility: Performance Review Examples (1 – 5)

Learn more:

How to Give Performance Feedback? Techniques and Examples (Positive, Negative, STAR Feedback)

More Performance Review Examples

Here you can find more performance review examples: click to open 2000+ Performance Review Phrases: The Complete List

  • Job Knowledge Performance Review Phrases (Examples)
  • How to Write an Employee Evaluation [Examples]
  • 30 Employee Feedback Examples (Positive & Negative)
  • Quantity Of Work Examples For Performance Reviews (Full Guide)
  • How to Give Performance Feedback? Techniques and Examples (Star Feedback)
  • How To Write a Manager Performance Review? (with Examples)

Initiative: 40 Useful Performance Feedback Phrases

Initiative: Use these sample phrases to craft meaningful performance evaluations, drive change and motivate your workforce.

An initiative is the ability to assess and initiate things independently often done without any managerial influence offered.

Initiative: Exceeds Expectations Phrases

  • Does not pause to be told what to do, always seeks new tasks to work on
  • Doesn't need close supervision in any functions, works well without any supervision
  • Always a goal oriented person sets priorities and accomplishes them
  • Highly flexible with the ability to adjust to the ever-changing needs of the company.
  • A highly creative person who is skilled in finding ways to perfect the performance
  • Has a high sense of responsibility always completing projects on time
  • Never feels satisfied with the performance, always seeking new ways to improve
  • A great supporter of new ideas, goals and working methods no matter who suggests them
  • Always ready to take on new tasks whenever it happens
  • Has excellent communication skills that get the job done through talking to others

Initiative: Meets Expectations Phrases

  • Takes the initiative in overcoming obstacles and meeting the organization's goals
  • Holds high ethical standards that are apparent through all the work delivered
  • Always goes above and beyond the job description and duties to satisfy the customers
  • Very compassionate and listens to the client's concerns making them know they are heard
  • Shows high energy in undertaking challenges related to the assigned tasks
  • Highly committed to maintaining punctuality that has contributed to a large extent to the overall success of the team
  • Always eager to obtain constructive feedback that ensures better performance in all the tasks
  • Instill a sense of teamwork among the co-workers promoting an atmosphere of cooperation and collaboration
  • Always willing to accept responsibility for the tasks given and remain accountable
  • Always ready to take on more tasks even before finishing the first

Initiative: Needs Improvement Phrases

  • Not as experienced and knowledgeable as shown in the application
  • Must always be under close supervision to perform or complete any task
  • Highly inactive and not ready to learn any new skills or develop older ones
  • A difficult person who is too difficult to handle generally or follow rules
  • Hardly improves in any given task and continues to neglect new opportunities presented
  • Does not perform under pressure and finds it hard to adjust to new or challenging atmosphere
  • Does not think outside the box and is always afraid of making concrete decisions
  • Does not generate any new ideas or recognize new solutions for problems
  • Has inferior abilities to establish personal priorities and courses of action
  • Lacks planning skills, therefore, does every task without a viable plan

Initiative: Self Evaluation Questions

  • Are you pushed to think creatively and explore new opportunities?
  • Do you have to be told what to do or you take the lead?
  • Are you creative enough to come up with decisions for hard tasks?
  • Have you sighted any improvement from the last evaluation season?
  • Are you prepared to learn new skills and incorporate them into your work?
  • Do you have the ability to plan your personal goals and accomplish them?
  • How well do you handle work pressure, deadlines, and coworkers?
  • Do you establish personal priorities to enable you to complete first tasks first?
  • Do you accept responsibility quickly without passing on the blame?
  • Do you take positive criticism well or do you allow your feeling to lead you?

These articles may interest you

Recent articles.

  • Employee Performance Goals Sample: Weather Analyst
  • Outstanding Employee Performance Feedback: Director of Social Services
  • Outstanding Employee Performance Feedback: Senior Cytogenetic Technologist
  • Poor Employee Performance Feedback: Chief Meteorologist
  • Employee Performance Goals Sample: Concrete Foreman
  • Smart Employee Performance Evaluation Phrases For Managers
  • Poor Employee Performance Feedback: Counseling Services Director
  • Employee Performance Goals Sample: Engineering Director
  • Outstanding Employee Performance Feedback: Customs Compliance Auditor
  • Employee Performance Goals Sample: Research Greenhouse Supervisor
  • Employee Performance Goals Sample: Operations Clerk
  • Planning and Scheduling: 15 Examples for Setting Performance Goals
  • Sample Employee Recognition Award Letter
  • Good Employee Performance Feedback: Data Infrastructure/Visualization Specialist
  • Employee Performance Goals Sample: Auxiliary Engineer

Our next-generation model: Gemini 1.5

Feb 15, 2024

The model delivers dramatically enhanced performance, with a breakthrough in long-context understanding across modalities.

SundarPichai_2x.jpg

A note from Google and Alphabet CEO Sundar Pichai:

Last week, we rolled out our most capable model, Gemini 1.0 Ultra, and took a significant step forward in making Google products more helpful, starting with Gemini Advanced . Today, developers and Cloud customers can begin building with 1.0 Ultra too — with our Gemini API in AI Studio and in Vertex AI .

Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. In fact, we’re ready to introduce the next generation: Gemini 1.5. It shows dramatic improvements across a number of dimensions and 1.5 Pro achieves comparable quality to 1.0 Ultra, while using less compute.

This new generation also delivers a breakthrough in long-context understanding. We’ve been able to significantly increase the amount of information our models can process — running up to 1 million tokens consistently, achieving the longest context window of any large-scale foundation model yet.

Longer context windows show us the promise of what is possible. They will enable entirely new capabilities and help developers build much more useful models and applications. We’re excited to offer a limited preview of this experimental feature to developers and enterprise customers. Demis shares more on capabilities, safety and availability below.

Introducing Gemini 1.5

By Demis Hassabis, CEO of Google DeepMind, on behalf of the Gemini team

This is an exciting time for AI. New advances in the field have the potential to make AI more helpful for billions of people over the coming years. Since introducing Gemini 1.0 , we’ve been testing, refining and enhancing its capabilities.

Today, we’re announcing our next-generation model: Gemini 1.5.

Gemini 1.5 delivers dramatically enhanced performance. It represents a step change in our approach, building upon research and engineering innovations across nearly every part of our foundation model development and infrastructure. This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture.

The first Gemini 1.5 model we’re releasing for early testing is Gemini 1.5 Pro. It’s a mid-size multimodal model, optimized for scaling across a wide-range of tasks, and performs at a similar level to 1.0 Ultra , our largest model to date. It also introduces a breakthrough experimental feature in long-context understanding.

Gemini 1.5 Pro comes with a standard 128,000 token context window. But starting today, a limited group of developers and enterprise customers can try it with a context window of up to 1 million tokens via AI Studio and Vertex AI in private preview.

As we roll out the full 1 million token context window, we’re actively working on optimizations to improve latency, reduce computational requirements and enhance the user experience. We’re excited for people to try this breakthrough capability, and we share more details on future availability below.

These continued advances in our next-generation models will open up new possibilities for people, developers and enterprises to create, discover and build using AI.

Context lengths of leading foundation models

Highly efficient architecture

Gemini 1.5 is built upon our leading research on Transformer and MoE architecture. While a traditional Transformer functions as one large neural network, MoE models are divided into smaller "expert” neural networks.

Depending on the type of input given, MoE models learn to selectively activate only the most relevant expert pathways in its neural network. This specialization massively enhances the model’s efficiency. Google has been an early adopter and pioneer of the MoE technique for deep learning through research such as Sparsely-Gated MoE , GShard-Transformer , Switch-Transformer, M4 and more.

Our latest innovations in model architecture allow Gemini 1.5 to learn complex tasks more quickly and maintain quality, while being more efficient to train and serve. These efficiencies are helping our teams iterate, train and deliver more advanced versions of Gemini faster than ever before, and we’re working on further optimizations.

Greater context, more helpful capabilities

An AI model’s “context window” is made up of tokens, which are the building blocks used for processing information. Tokens can be entire parts or subsections of words, images, videos, audio or code. The bigger a model’s context window, the more information it can take in and process in a given prompt — making its output more consistent, relevant and useful.

Through a series of machine learning innovations, we’ve increased 1.5 Pro’s context window capacity far beyond the original 32,000 tokens for Gemini 1.0. We can now run up to 1 million tokens in production.

This means 1.5 Pro can process vast amounts of information in one go — including 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code or over 700,000 words. In our research, we’ve also successfully tested up to 10 million tokens.

Complex reasoning about vast amounts of information

1.5 Pro can seamlessly analyze, classify and summarize large amounts of content within a given prompt. For example, when given the 402-page transcripts from Apollo 11’s mission to the moon, it can reason about conversations, events and details found across the document.

Reasoning across a 402-page transcript: Gemini 1.5 Pro Demo

Gemini 1.5 Pro can understand, reason about and identify curious details in the 402-page transcripts from Apollo 11’s mission to the moon.

Better understanding and reasoning across modalities

1.5 Pro can perform highly-sophisticated understanding and reasoning tasks for different modalities, including video. For instance, when given a 44-minute silent Buster Keaton movie , the model can accurately analyze various plot points and events, and even reason about small details in the movie that could easily be missed.

Multimodal prompting with a 44-minute movie: Gemini 1.5 Pro Demo

Gemini 1.5 Pro can identify a scene in a 44-minute silent Buster Keaton movie when given a simple line drawing as reference material for a real-life object.

Relevant problem-solving with longer blocks of code

1.5 Pro can perform more relevant problem-solving tasks across longer blocks of code. When given a prompt with more than 100,000 lines of code, it can better reason across examples, suggest helpful modifications and give explanations about how different parts of the code works.

Problem solving across 100,633 lines of code | Gemini 1.5 Pro Demo

Gemini 1.5 Pro can reason across 100,000 lines of code giving helpful solutions, modifications and explanations.

Enhanced performance

When tested on a comprehensive panel of text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing our large language models (LLMs). And when compared to 1.0 Ultra on the same benchmarks, it performs at a broadly similar level.

Gemini 1.5 Pro maintains high levels of performance even as its context window increases. In the Needle In A Haystack (NIAH) evaluation, where a small piece of text containing a particular fact or statement is purposely placed within a long block of text, 1.5 Pro found the embedded text 99% of the time, in blocks of data as long as 1 million tokens.

Gemini 1.5 Pro also shows impressive “in-context learning” skills, meaning that it can learn a new skill from information given in a long prompt, without needing additional fine-tuning. We tested this skill on the Machine Translation from One Book (MTOB) benchmark, which shows how well the model learns from information it’s never seen before. When given a grammar manual for Kalamang , a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content.

As 1.5 Pro’s long context window is the first of its kind among large-scale models, we’re continuously developing new evaluations and benchmarks for testing its novel capabilities.

For more details, see our Gemini 1.5 Pro technical report .

Extensive ethics and safety testing

In line with our AI Principles and robust safety policies, we’re ensuring our models undergo extensive ethics and safety tests. We then integrate these research learnings into our governance processes and model development and evaluations to continuously improve our AI systems.

Since introducing 1.0 Ultra in December, our teams have continued refining the model, making it safer for a wider release. We’ve also conducted novel research on safety risks and developed red-teaming techniques to test for a range of potential harms.

In advance of releasing 1.5 Pro, we've taken the same approach to responsible deployment as we did for our Gemini 1.0 models, conducting extensive evaluations across areas including content safety and representational harms, and will continue to expand this testing. Beyond this, we’re developing further tests that account for the novel long-context capabilities of 1.5 Pro.

Build and experiment with Gemini models

We’re committed to bringing each new generation of Gemini models to billions of people, developers and enterprises around the world responsibly.

Starting today, we’re offering a limited preview of 1.5 Pro to developers and enterprise customers via AI Studio and Vertex AI . Read more about this on our Google for Developers blog and Google Cloud blog .

We’ll introduce 1.5 Pro with a standard 128,000 token context window when the model is ready for a wider release. Coming soon, we plan to introduce pricing tiers that start at the standard 128,000 context window and scale up to 1 million tokens, as we improve the model.

Early testers can try the 1 million token context window at no cost during the testing period, though they should expect longer latency times with this experimental feature. Significant improvements in speed are also on the horizon.

Developers interested in testing 1.5 Pro can sign up now in AI Studio, while enterprise customers can reach out to their Vertex AI account team.

Learn more about Gemini’s capabilities and see how it works .

Get more stories from Google in your inbox.

Your information will be used in accordance with Google's privacy policy.

Done. Just one step more.

Check your inbox to confirm your subscription.

You are already subscribed to our newsletter.

You can also subscribe with a different email address .

Related stories

What is a long context window.

MSC_Keyword_Cover (3)

How AI can strengthen digital security

Shield

Working together to address AI risks and opportunities at MSC

AI Evergreen 1 (1)

How we’re partnering with the industry, governments and civil society to advance AI

NWSL_Pixel_Hero

Pixel is now the Official Mobile Phone of the National Women’s Soccer League

Bard_Gemini_Hero

Bard becomes Gemini: Try Ultra 1.0 and a new mobile app today

Let’s stay in touch. Get the latest news from Google in your inbox.

  • Open access
  • Published: 20 February 2024

The role of provincial health administration in supporting district health management teams in the Democratic Republic of Congo: eliciting an initial programme theory of a realist evaluation

  • Samuel Bosongo   ORCID: orcid.org/0000-0001-8097-6721 1 , 2 , 3 , 4 ,
  • Zakaria Belrhiti 5 , 6 ,
  • Faustin Chenge 1 , 4 , 7 ,
  • Bart Criel 3 ,
  • Yves Coppieters 2 &
  • Bruno Marchal 3  

Health Research Policy and Systems volume  22 , Article number:  29 ( 2024 ) Cite this article

Metrics details

In 2006, the Ministry of Health in the Democratic Republic of Congo designed a strategy to strengthen the health system by developing health districts. This strategy included a reform of the provincial health administration to provide effective technical support to district health management teams in terms of leadership and management. The provincial health teams were set up in 2014, but few studies have been done on how, for whom, and under what circumstances their support to the districts works. We report on the development of an initial programme theory that is the first step of a realist evaluation seeking to address this knowledge gap.

To inform the initial programme theory, we collected data through a scoping review of primary studies on leadership or management capacity building of district health managers in sub-Saharan Africa, a review of policy documents and interviews with the programme designers. We then conducted a two-step data analysis: first, identification of intervention features, context, actors, mechanisms and outcomes through thematic content analysis, and second, formulation of intervention–context–actor–mechanism–outcome (ICAMO) configurations using a retroductive approach.

We identified six ICAMO configurations explaining how effective technical support (i.e. personalised, problem-solving centred and reflection-stimulating) may improve the competencies of the members of district health management teams by activating a series of mechanisms (including positive perceived relevance of the support, positive perceived credibility of provincial health administration staff, trust in provincial health administration staff, psychological safety, reflexivity, self-efficacy and perceived autonomy) under specific contextual conditions (including enabling learning environment, integration of vertical programmes, competent public health administration staff, optimal decision space, supportive work conditions, availability of resources and absence of negative political influences).

Conclusions

We identified initial ICAMO configurations that explain how provincial health administration technical support for district health management teams is expected to work, for whom and under what conditions. These ICAMO configurations will be tested in subsequent empirical studies.

Peer Review reports

Introduction

In an ever-evolving world, health systems are under pressure. They have to perform better to meet the population’s expectations while dealing with various challenges, such as the ageing population, climate change, the double burden of infectious and non-communicable diseases, re-emerging diseases and violent conflicts [ 1 ]. However, health systems in most sub-Saharan countries remain weak and fragile, and they struggle to progress towards universal health coverage [ 2 ]. While the WHO acknowledges that health system strengthening is the principal means to achieve universal health coverage [ 3 ], little is known about how best to do so [ 2 ]. Decentralisation is a widespread health sector reform in sub-Saharan Africa that aims to improve health systems’ performance in terms of access, quality, equity, efficiency and financial protection [ 2 , 4 , 5 ]. However, better information and evidence are still needed to guide this reform [ 2 ].

The health system in the Democratic Republic of Congo (DRC)—whose overall structure is outlined in Box  1 —has been ranked among the worst-performing in Africa for almost three decades [ 6 , 7 , 8 ]. This situation is partly due to persistent socio-political crises and systematic underfunding of the health sector [ 9 ]. In response to these crises, a range of emergency interventions have been implemented by humanitarian agencies, and much space was given to vertical disease control programmes. However, instead of strengthening a comprehensive primary health care system, the foundation of the DRC’s national health policy, these programmes were selective and often a source of disruption and distortion of the already weakened regular health system [ 9 , 10 , 11 ]. Furthermore, a massive expansion of the number of universities led to an uncontrolled increase in medical doctors [ 12 ] and a booming but poorly regulated private-for-profit sector [ 9 , 10 ]. This situation contributed to increasing the disintegration, fragmentation, lack of coordination and inefficiency of the health system [ 9 ].

Box 1. The overall governance structure of the health system in the DRC

The health system in the DRC is structured into three levels: national, provincial and operational. At the national level, the National Ministry of Health oversees the General Secretariat for Health and the General Health Inspectorate, each with its central directorates. They are responsible for setting norms, policies, and guidelines and monitoring their implementation at the sub-national levels.

The provincial level includes the Provincial Ministry of Health, the Provincial Health Division (referred to in this paper as Provincial Health Administration) and the Provincial Health Inspectorate, each with its own offices. The provincial health division provides technical support to health districts, while the provincial health inspectorate ensures the enforcement of national-level norms, policies and guidelines.

The operational level consists of health districts where national health policies are implemented. It comprises two specific yet complementary healthcare levels, overseen by the district health management team. The first level includes a network of first-line health facilities that provide primary care to the population. The second level comprises one or more hospitals that offer more technical or specialised care. The district health management team includes healthcare professionals with management or administrative positions. They have diverse professional backgrounds, including physicians, nurses, pharmacists, nutritionists and administrators, and perform different roles, such as district medical officers, hospital directors, clinicians, nursing officers and nurse supervisors.

These three levels are linked hierarchically so that the lower levels are accountable to the direct higher level. Although the provincial and operational levels are supposed to operate in a decentralised fashion, they need more decision space, especially in human resource management and financial resource mobilisation and allocation.

In response to this situation, the Ministry of Health (MoH) developed the Health System Strengthening Strategy in 2006 [ 11 ] and updated it in 2010 [ 10 ]. This strategy followed the adoption of the new constitution, which embodies the principle of decentralisation [ 13 ]. A key pillar of this strategy is strengthening the health districts, considered the essential lever for strengthening the health system [ 10 , 11 ]. It recommended restructuring the Provincial Health Administration (PHA) so that it can provide effective technical support to district health management teams (DHMTs) to develop their leadership and management capacities. The PHA reform involved a functional, a structural and a cultural reorganisation. The functional reorganisation separated the inspection and control function from technical support to the health districts. The structural reorganisation involved moving from 13 offices and multiple vertical programmes to four core functions corresponding to four PHA offices: (1) technical support for health districts office, (2) health information, communication and research office, (3) inspection and control office and (4) resource management office. Two offices were added: the public hygiene office and the health sciences education office. In addition to these administrative offices, the PHA office also includes a number of thematic working groups. They are ad hoc functional bodies (or taskforces) where PHA staff from all offices can meet to discuss and reflect on specific issues, such as technical support to health districts, health information management, medicine supplies to health districts, health financing coordination and epidemiological surveillance. They are designed to promote synergy among staff, encourage participation in decision-making and foster learning. The cultural shift aimed to gradually transition from a hierarchical to an adhocracy culture [ 14 ].

Technical support to health districts is the central role of the PHA office. It is supposed to enhance the leadership and management capacities of DHMT members to improve the health district’s performance and, ultimately, the overall health outcomes of the population. These capacities include coordinating stakeholders, planning and budgeting, monitoring and evaluation, training and supervising health workers, managing health system information, conducting epidemiological surveillance, managing resources (human, financial, material and medicines) and conducting operational and action research [ 15 ]. Technical support is to be provided through facilitative supervision, coaching and problem-solving support. The PHA reform assigned the technical support role to experienced PHA staff with public health and district management backgrounds. Their number depends on the number of health districts in the province and the availability of qualified staff at the PHA offices. In practice, each PHA staff member is responsible for supporting two to four health districts. They provide technical support to DHMT members through field visits, which are scheduled based on issues identified by PHA staff through analysis of data, plans and activity reports of each health district or based on concerns raised by the DHMT members.

Between 2008 and 2011, pilot action research conducted in the provinces of North Kivu and Eastern Kasai to test PHA reorganisation pointed to positive results in terms of an adequate structure of PHA for providing better support to health districts [ 14 ]. Consequently, the PHA reform was rolled out in all 26 provinces of the DRC between 2014 and 2015. Since the rollout of the PHA reform, two studies on the technical support of DHMT members have reported contrasting results. One study found that DHMT members greatly appreciated technical support in one province [ 16 ]. However, in another province, there was a reported lack of a clear conceptual model to guide the operationalisation of this support [ 17 ]. This divergence may be partially explained by the research methods used (quantitative versus mixed methods) and the study contexts (rural and urban versus urban only). However, it also points to how technical support is being implemented across different provinces and how actors’ perceptions and responses influence this implementation process.

As a capacity building intervention, the technical support from the reformed PHAs to DHMTs is a complex intervention in districts which can be considered as complex systems [ 18 ]. In such systems characterised by continuous interactions between actors, their organisation and their environment, the outcomes (improved leadership and management capacities of DHMT members) of the technical support from the PHAs (intervention) cannot fully be predicted. Moreover, more knowledge about how such capacity building interventions improve the performance of health workers in low- and middle-income countries, such as the DRC, is needed [ 18 ]. In this paper, we present how we have elicited the initial programme theory of technical support, which is the first step of the realist evaluation we are conducting (explained below).

Methodology

The methodological approach.

Our overall methodological approach is realist evaluation (RE). RE is a theory-driven evaluation approach that seeks to understand why and how a programme works, for whom, and in what circumstances [ 19 , 20 ]. By answering these questions, RE attempts to provide a plausible causal account of how the interaction of actors with an intervention triggers mechanisms that lead to outcomes within a given context. This context-sensitive approach is well suited for evaluating complex interventions, such as capacity building within complex systems (for example health districts) [ 18 ].

The realist approach considers social programmes to be theories, active and embedded in social systems [ 20 ]. This view of social programmes has methodological implications for RE. First, as programmes are theories incarnate, realist researchers are tasked with eliciting, testing and refining the underlying programme theories [ 20 ]. A programme theory is a set of assumptions explaining how an intervention brings about changes (intended or not) by activating mechanisms among actors in a given context. Second, programmes work through people’s reasoning. In other words, programmes do not bring about changes but the actors do through their reasoning and responses to the resources provided by a programme. People’s reasoning and resources are called mechanisms and generate outcomes in a particular context [ 21 , 22 , 23 ]. Hence, realist evaluators seek to identify these underlying generative mechanisms. Finally, programmes are open systems embedded in social systems that continuously interact and influence each other. Thus, an essential requirement of RE is to take heed of the social environment (or context) surrounding programmes as it conditions the firing of mechanisms [ 20 , 24 , 25 , 26 ].

In practice, RE begins and ends with a theory [ 27 , 28 ]. It has three main stages: (1) eliciting the initial programme theory (IPT), (2) testing the IPT through empirical studies and (3) refining the IPT based on the results of empirical studies. The resulting refined programme theory is to be further tested and refined in new studies. Realist researchers use the context–mechanism–outcome (CMO) configuration as a heuristic tool. In this study, we use a fine-tuned variant, the intervention–context–actors–mechanisms–outcomes (ICAMO) configuration, to better differentiate intervention from context and emphasise the role of different actors in the change processes [ 29 ]. In realist evaluation literature, intervention and context are sometimes conflated to the extent that aspects of ‘intervention’ are often reported as ‘contextual factors’. However, context and intervention are separate conceptual and analytical entities in realist evaluation. Therefore, it is important to provide a detailed description of contextual factors and intervention components or features. This will help to identify which contextual factors influence which intervention component, thus triggering the mechanisms that lead to the outcomes. Furthermore, when implementing an intervention, it is essential to recognize that various stakeholders have unique roles, perspectives and reasoning. Emphasizing these differences can provide valuable insights into the ‘for whom’ question in realist evaluation, and the ICAMO configuration is a heuristic that draws the attention of the analysts to these issues [ 30 ].

Data collection

We collected data through a scoping review of the capacity building of district health managers in sub-Saharan Africa [ 31 ], a review of documents related to the PHA reform in the DRC and in-depth interviews with stakeholders of the PHA reform (Fig.  1 ).

figure 1

The process of IPT development

Scoping review

We conducted a scoping review to describe how capacity building programmes for district health managers are designed, delivered and evaluated in sub-Saharan Africa. We focused on identifying the underlying assumptions or theories behind these programmes. We searched for relevant studies through five electronic databases (PubMed, Health Systems Evidence, Wiley Online Library, Cochrane Library and Google Scholar), grey literature and citation tracking. We included all primary studies reporting leadership or management capacity building of district health managers in sub-Saharan Africa, written in English or French, and published between 1 January 1987 and 13 October 2022. Further details on the scoping review can be found elsewhere [ 31 ].

Document review

We aim to understand the process of PHA reform in the DRC and to collect general information about the implicit logic model of the technical support from PHA staff to DHMT members. The documents were from different sources, including the MoH, supporting partner organisations and previous studies in the DRC. We included 21 documents based on their relevance, i.e. documents that provide appropriate information related to the PHA reform and technical support to health districts [ 32 , 33 ]. The type and description of the included documents are summarised in the Additional file 1 .

In-depth interviews

To gain a better understanding of how and why the intervention would bring about expected changes, we conducted interviews with stakeholders involved in the design of the PHA reform (Table  1 ). We purposively identified 15 potential respondents and contacted them via email to invite them for an interview. An information sheet on the study was provided. Thirteen participants accepted, and ten respondents were interviewed. Three accepted but could not be interviewed due to persistently conflicting agendas. We used a piloted interview guide with open-ended questions (Additional file 2 ). Questions were related to the general process of PHA reform in the DRC, the expected outcomes from the reform and the technical support, the process of technical support and the contextual factors that may influence the technical support process and outcomes. All interviews were conducted in French by the first author (a male medical doctor from the DRC and a PhD student trained in qualitative methods) and online using Teams, Zoom and WhatsApp applications in March 2023. Nine interviews were audio recorded and lasted 50 min on average, and one interview was conducted through WhatsApp chat due to poor internet connection. The interviews were transcribed verbatim and sent to the participants for comment and/or correction. In addition to audio-recorded interviews, we sent follow-up questions via email to four respondents to gain a deeper understanding of certain issues that arose during our data analysis. Saturation was achieved after the ten interviews and follow-up questions. Therefore, no further interviews were conducted with other potential participants identified to replace those who were not available due to persistent conflicting agendas.

Data analysis

The analysis of the data from the three sources was combined. We analysed the data in two steps: (1) identification of ICAMO components and (2) formulation of ICAMO configurations (Fig.  2 ).

figure 2

Data analysis process

Step 1. Identification of ICAMO components

To manage the data, we imported the interview transcripts, documents, and results of the scoping review into N-Vivo 14. After familiarisation with the transcripts and the documents through multiple readings, the first author performed a framework analysis [ 34 ], applying thematic content analysis to identify themes related to the intervention, context, actors, mechanisms and outcomes. We performed both manifest and latent content analysis [ 35 ]. We started the coding with a focus on the manifest content, focusing on the terms and concepts used by the respondents to describe elements of intervention, context, actors, mechanisms and outcomes. In a second reading of the interviews, we focused on the latent content, probing for the interpretations of the respondents, and more specifically on whether and how they identified causal explanations. An a priori codebook developed by the research team was used (Table  2 ) [ 32 ]. The following questions guided this step:

What are the expected outcomes of the technical support of DHMT members by PHA staff?

What are the components or features of this technical support? How should these components be carried out?

Who are the actors involved in implementing technical support?

What contextual factors can facilitate or hinder actors in taking up technical support?

What possible mechanisms can be triggered by technical support for producing the reported outcomes?

Step 2. Formulation of ICAMO configurations

In the second step, we used a retroductive approach to identify the links between the intervention, context, actors, mechanisms and outcomes, i.e. the ICAMO configurations. The ICAMO configuration is a plausible causal pathway that explains how actors deal with intervention components within a given context, which activates mechanisms that lead to outcomes. Retroduction is a mode of inference that seeks to ‘unearth the activated mechanisms’ [ 25 , 26 ]. In practice, we started from the expected outcomes and worked backwards through data to determine the intervention component, plausible mechanisms and contextual conditions that could cause them. The guiding questions at this step were as follows:

What outcome(s) can be linked to the implementation of the component(s) of technical support of DHMT members by PHA staff?

What mechanism(s) can link the outcome(s) to the component(s) of technical support of DHMT members by PHA staff?

What contextual condition(s) can facilitate or hinder the activation of such mechanism(s)?

Are there alternative explanations?

Ethical considerations

The research protocol was approved by the Institutional Review Board (IRB) of the Institute of Tropical Medicine, Antwerp (reference IRB no. 1654/22) and the Medical Ethics Committee of the University of Lubumbashi (reference no. UNILU/CEM/005/2023). Prior to the interviews, we sent the study information sheet and the informed consent form to the potential participants and obtained their agreement by e-mail. The information sheet contained information about the study’s objectives, the voluntary nature of the participation, confidentiality measures and benefits and risks associated with the study. To ensure confidentiality, we pseudonymised the data and gave each participant a code. All data were stored in a password-protected drive.

In this section, we first present the individual components of ICAMO, followed by the ICAMO configurations.

ICAMO components

In this subsection, we begin with the expected outcomes, the starting point of our retroductive approach described above.

The expected immediate outcome is a competent DHMT. Technical support to health districts is meant to strengthen the competencies of DHMT members in performing their managerial tasks (coordination, planning, monitoring, evaluation, supervision, managing resources, etc.) and clinical functions, as explained in the excerpts below:

“The PHA staff should strengthen the DHMT members’ managerial functions in order to enable them to develop a functional local health system”. [DR 12 ].
“The clinical function is an essential aspect that is often overlooked, yet this is the raison d'être of any healthcare system. A DHMT must have clinical competencies to supervise health facilities effectively”. [IDI 1 ]

The intermediate outcome is to improve the performance of health districts. A competent DHMT is expected to improve the performance of its district. This includes optimising the health district as an integrated system and improving the coverage, access, equity and quality of health care and services.

“The purpose of technical support to health districts is to enhance the quality, accessibility, and coverage of health care and services through integrated leadership of the DHMT, which connects health centres and referral hospitals while encouraging the integration of vertical programmes and community participation”. [DR 7 ]

In terms of long-term outcomes, an improved population health status is expected. This aligns with the general objective of the DRC’s national health development plan:

“[…] to enhance the overall health of individuals, allowing them to lead a healthy life, and promote well-being for all, regardless of age”. [DR 14 ]

Intervention

We found that the intervention includes actions at the provincial and district levels.

At the provincial level, actions aim to enhance the abilities of PHA staff to offer effective technical support to DHMT members. These include training sessions and regular meetings of PHA staff.

The training of PHA staff should emphasise action. This aligns with the scoping review’s findings, which identified the ‘action learning or learning-by-doing approach’ [ScR] as a key feature of effective capacity building programmes for district health managers. This is also echoed in the quote below:

“One option to help PHA staff learn their job is to pair them up with an experienced colleague or advisor. This technique of know-how transfer seems the only effective one”. [IDI 2 ]

Regular meetings of PHA staff are conversational spaces for PHA staff to discuss technical support issues, share their field experiences and learn from each other:

“Another option [for strengthening the competencies of PHA staff] is to share their experiences. I think there are structures that allow the PHA staff to get together, such as the working groups, the provincial health management team and others. In principle, these meetings should be regular and focus on discussing specific technical support issues, proposing solutions, evaluating their effectiveness, and learning. This may enhance their [PHA staff] competencies through practical experience”. [IDI 2 ]

At the district level, technical support for health districts consists of “strengthening the capacities of the DHMT members and healthcare providers […] through training, supportive supervision, problem-solving support, health data analysis, and guidance on health policies and guidelines”. [DR 7 ] The terms ‘formative supervision’ and ‘coaching’ are used interchangeably to describe this support. Both focus on “shifting from an administrative and prescriptive approach to a supportive and formative one”. [DR 18 ] The key features of this approach are being personalised, problem-solving-centred, reflection-stimulating, regular and continuous and comprehensive.

Personalised support is about providing adequate support for DHMT members, which aligns with their current needs. Involving DHMT members in identifying their own support needs is crucial, as highlighted in the quote below:

“Since coaching is personalised, it should be tailored to the team's or individual's needs; coaching cannot be envisaged on issues decided unilaterally by the coach”. [DR 13 ]

Problem-solving support-solving support is an essential component of technical support for DHMT members. PHA staff are expected to have problem-solving skills, as echoed by this informant:

“PHA staff should be able to identify problems, work with DHMT members to find solutions, and provide guidance and adjustments as needed while avoiding taking over the DHMT’s responsibilities”. [IDI 9 ]

Reflection-stimulating support enables DHMT members to reflect on and learn from their practices and performances. In such support, the role of the PHA staff in asking the right questions to stimulate reflection within the DHMT is crucial:

“Asking questions, such as ‘why this?’ and ‘why that?’ can help people [DHMT members] reflect on and potentially improve their work. This spirit of reflexivity is often lacking in DHMT members and should be encouraged”. [IDI 8 ]

Regular and continuous support is important. Technical support missions should last ‘at least one week’ [IDI 3 ] and occur ‘at least once a quarter’. [IDI 6 ] The support must be ‘regular and continuous’ [DR 18 ] to blend in with the team and ensure a smooth transfer of know-how. Technical support extends beyond field missions, as this informant points out:

“It [technical support] is permanent work with two parts: working in person with frequent and extended visits to the district and working remotely from the PHA office to support the health district. When we were drafting this [PHA] reform, we did not have access to tools like Zoom or WhatsApp, but now we have many more options for improving remote communication during this permanent work”. [IDI 6 ]

Comprehensive support: technical support for health districts is intended to cover all aspects of health district development. However, the PHA staff should induce vertical supervision for issues for which they do not have the required skills.

“A PHA staff should take a comprehensive approach to developing the health district to avoid fragmentation. They must also recognise when more than their own skills are needed to meet the needs of DHMT members and bring in additional expertise as needed”. [DR 13 ]

We organised the contextual factors into the national, provincial and district levels.

At the national level, support from the national MoH is a condition for the success of the provincial-level reform and, thus, technical support for the health districts.

“The success of such restructuring [PHA reform] depends on the support and supervision from the national Ministry of Public Health”. [DR 7 ]

The National Health Development Plan 2016–2020 recognised that “the un-reformed national Ministry of Health was not providing enough support for the PHA reform”. [DR1 4 ] This is also underlined by one informant in the following terms:

“[...] even at the central level, I get the impression that each of its many departments is working for itself [...], so there is a major lack of harmonisation and coherence in the institutional system that does not make easier the support of the reformed PHA from the central level”. [IDI 2 ]

To effectively support PHA, the National Health Development Plan 2016–2020 emphasised the importance of ‘accelerating the reform of the national Ministry of Health’. [DR1 4 ]

At the provincial level, we identified two major themes: the optimal functioning of the PHA office and support from the provincial political leaders in the context of decentralisation.

The optimal functioning of the PHA office is a prerequisite for effective technical support to health districts. This implies harmonious coordination among its offices, effective leadership, integration of vertical programmes, availability of resources and an optimal decision-making space.

Harmonious coordination among the various PHA offices is achieved through thematic working groups. These are supposed to facilitate teamwork, break down communication barriers, and enhance participative decision-making and individual and collective learning. They add an adhocratic dimension to PHA functioning.

“It [PHA] is a structure that must stop functioning as a pure administration, leave behind the bureaucratic model and migrate towards a different model, [...] It must function as teamwork without compartmentalisation between offices”. [IDI 5 ]

Effective leadership is essential for the optimal functioning of the PHA. This involves a clear and shared vision of the role of the PHA and a ‘willingness to implement reforms’. [DR 12 ] Without these, there is a risk of ‘reproducing a dysfunctional system’. [IDI 2 ].

“The PHA requires strong leadership [...]. The head of the PHA office should have a clear vision of the expected role of PHA according to the Health System Strengthening Strategy and share it with the staff [...]. This will enable the PHA office to fulfil its mission and play its role more effectively if there is the will”. [IDI 2 ]

An optimal level of administrative integration of (vertical) disease control programmes may enable the capture of their financial resources (which ‘account for 60% of financial resources’ [DR 16 ]) and better coordinate technical support for health districts, thus preventing overlapping activities.

“The overlap [of technical support] with other activities could be explained by the fact that the funding and actions of specialised programmes still need to be sufficiently integrated at the PHA”. [DR 18 ]

Disease control programs rely primarily on external funding, so their integration requires the alignment of funders with national health policies and priorities of PHA. The lack of this alignment may contribute to persistent dysfunction of the health system, as described by this informant:

“Some partners do not align with the national policy or health system strengthening strategy. Due to their financial power, they can influence political decisions and cause disruptions in healthcare organisations. Unfortunately, public funds are limited, and only some activities are funded by those with resources, sometimes at the expense of prioritising the development of health districts in line with the health system strengthening strategy”. [IDI 2 ]

Furthermore, the integration of disease control programmes at the national MoH was identified as a prerequisite for their integration at the PHA:

“Integrating specialised programmes at the provincial level may only be successful with national-level reflection and action in this direction”. [DR 19 ]

The availability of resources is crucial for the optimal functioning of the PHA office. These resources include first competent and sufficient (number of) human resources to cover all the health districts.

“The PHA must first have the right human resources, i.e. people who are competent and morally upright in sufficient numbers to cover all the health districts”. [IDI 1 ]

In addition to the quality and quantity of human resources, adequate financial, material and infrastructural resources are essential to guarantee optimal working and living conditions for PHA staff.

“The biggest issue lies in the precarious living conditions of health workers. Human beings play a crucial role in providing technical support, and to perform their job effectively, they require optimal working and living conditions”. [IDI 2 ]

The optimal functioning of the PHA office also requires an optimal decision space for PHA leaders to make the needed decisions and be shielded from harmful influences at the national or provincial level. A participant described the negative influence of national level and political leaders in the following terms:

“They [PHA leaders] do not have the autonomy they should have. They are sometimes influenced by the national level or the provincial authority, which they must continue to satisfy”. [IDI 2 ]

Support from the provincial political leaders: In a decentralisation context, certain matters, such as the ‘promotion and organisation of primary health care’ [DR 20 ], fall under the exclusive competence of the provinces. Support from provincial political leaders is essential for the success of PHA reform and, thus, technical support for health districts.

“You know, the province has some degree of decentralisation, which means that it has certain responsibilities. Therefore, it is important to have a Governor and a [Provincial] Minister [of Health] who understand and support the PHA in achieving its goals”. [IDI 1 ]

The scoping review also identified ‘support from and collaboration with the government authorities’ [ScR] as a factor in the success of capacity building interventions for DHMT members.

At the district level, the contextual factors identified are the leadership and decision space of the DHMT, the working environment and the availability of resources.

The leadership and decision space of DHMT are important for health districts, as noted in the health system strengthening strategy:

“The success of developing health districts relies heavily on the leadership of DHMT. Hence, the DHMT must have a shared vision of the health district's development and the autonomy to make necessary decisions in response to identified problems”. [DR 1 ]

According to our scoping review, one of the success factors for capacity building interventions for DHMT members is ‘distributed leadership and the role of the head of health district, who can act as a local champion’. [ScR].

One informant stressed the importance of regular meetings within the DHMT in the following terms:

“It is important to regularly schedule meetings for sharing information and holding each team member accountable for their responsibilities. These meetings also allow the team to acknowledge each other’s contributions and find ways to work together more effectively”. [IDI 5 ]

An adequate working environment is essential for the performance of health districts. The informants defined the appropriate working environment as one that ‘offers the right working conditions’ [IDI 2 ] and one that ‘offers the necessary resources to carry out the various tasks of the district management team, without outside interferences, especially political interferences’. [IDI 1 ] Beyond this material dimension, the scoping review identified the human dimension of the working environment in terms of ‘safe climate work, supportive relationships, teamwork’. [ScR] One informant warns in the following terms:

“The working environment of the DHMT should be noticed. Neglecting to improve it could [negatively] affect the acceptance of technical support and collaboration with PHA staff”. [IDI 4 ]

The availability of adequate resources at the district level is crucial for enhancing its performance. These resources include competent human resources and adequate material and financial resources:

“[It is] impossible to develop a health district that is not financed”. [DR 1 ]

We identified two categories of key actors: PHA staff and DHMT members.

PHA staff are the providers of technical support to DHMT members. They need professional experience, a gradient of competencies and a positive posture to provide effective technical support to DHMT members.

The PHA staff’s professional experience is important for effective technical support for DHMT members. Indeed, the PHA staff should have ‘successful work experience at the health district level’ [IDI 5 ] and ‘useful experience related to the areas they are supporting’ [DR 7 ] so that ‘they can use them to support DHMT members in problem-solving’. [IDI 9 ] Otherwise, ‘they may only have theoretical knowledge and lack practical reference points’. [IDI 4 ].

The PHA staff’s competencies also matter in the technical support process. The PHA staff must have ‘a higher gradient of competencies’ [DR15] than the DHMT members for effective technical support.

“He [PHA staff] must be someone with a higher gradient of competencies than the DHMT members. It would not be appropriate to bring in the health district a PHA staff with a lower level of competencies than the head of the health district or the director of the district hospital, for instance”. [IDI 6 ]

We classified the PHA staff's competencies into knowledge, know-how and interpersonal skills. The knowledge and know-how of PHA staff refer to their abilities in management, clinical work and facilitation:

“The PHA staff must have sufficient adequate knowledge of the organisation and functioning of the national health system, policies, strategies and directives, the management of both the health system and health districts, as well as clinical practice”. [DR 7 ]

The PHA staff’s interpersonal skills refer to ‘relational qualities made up of a series of attitudes that enable the development of positive and harmonious social relationships’. [DR 12 ] The attitudes of good PHA staff that emerged from the interviews and document review include ‘empathy’ [IDI 1 , IDI 6 , DR 7 , DR 12 ], ‘listening” [IDI 4 , DR 18 , DR 7 , DR 9 , DR 18 ], ‘open-mindedness’ [IDI 1 , IDI 2 , DR 15 ], ‘knowing how to communicate or dialogue’ [IDI 1 , IDI 2 , IDI 4 , IDI 5 , DR 7 , DR 9 , DR 15 ], ‘observation skills’ [IDI 4 , IDI 8 ], ‘humility or modesty’ [IDI 2 , DR 7 ], ‘good character or moral probity’ [IDI 1 , DR 7 ] and ‘being available’ [IDI 6 , DR 7 , DR 9 , DR 12 ].

The PHA staff’s posture is crucial in determining the quality of interactions with DHMT members. Informants agree that PHA staff should avoid being hierarchical and instead adopt a coaching posture to encourage reflection within DHMT.

“In the technical support process, hierarchical posture can bias relationships and hinder the empowerment of individuals and learning processes [...]. Therefore, PHA staff should adopt a professional coach's posture, where they maintain an equal position with the coached team members while still being able to question practices and dynamics. By doing so, the coach becomes an initiator, catalyst, and companion in strengthening the individual and collective competencies, promoting innovation and implementing necessary changes”. [DR 13 ]

DHMT members directly benefit from technical support from PHA staff and are key players in developing health districts. DHMTs are ‘interdisciplinary and multiskilled teams’ [DR 4 ] and ‘responsible for managing the entire health district’. [DR 1 ] The compendium of standards for the organisation and functioning of health districts in the DRC states that the composition of DHMT can vary. However, the members must meet the following criteria: 1) be ‘people capable of working in a team and interested in the dynamic structuring of a health district functioning as an integrated health system’; 2) have ‘a gradient of competencies (acquired through qualification or experience) with staff who are not members of the district management team. Otherwise, supervision is no longer acceptable’; and 3) ‘have broader skills to translate into managerial terms the observations noted in patient management’. [DR 8 ].

We identified mechanisms for the PHA staff and DHMT members. Box  2 presents our definitions.

Box 2. Definition of psychological mechanisms identified

Self-efficacy refers to the belief of an individual in their ability to perform specific actions that lead to achieving certain goals [ 37 , 38 , 39 ].

Motivation is the process through which a person is stimulated to act. Motivation can be extrinsic or intrinsic. Extrinsic motivation is driven by the expectation of receiving a reward or avoiding punishment for performing an activity, while intrinsic motivation stems from an individual’s genuine interest in the activity itself and their ability to derive personal satisfaction from it [ 40 , 41 , 42 ].

Psychological safety is a belief shared by individuals about whether it is safe to take interpersonal risks in the workplace [ 43 ]. This risk-taking involves speaking up to voice ideas or challenge the status quo without fear of embarrassment, punishment, marginalisation or humiliation [ 44 ].

Reflexivity refers to the extent to which a person or a team actively reflects upon their (past) analyses, decisions and actions and how this may lead to adapting them as needed based on current or anticipated circumstances [ 45 , 46 ].

Trust is a psychological state involving acceptance of vulnerability based on positive expectations of another’s intentions or behaviour [ 47 ].

Autonomy is a basic psychological need that means having an optimal degree of freedom and control over one’s actions [ 40 , 41 , 42 ].

Mechanisms for the PHA staff

Self-efficacy was identified as a key mechanism by PHA staff. They gain skills through training and exchange of experience, which enhance their self-efficacy to effectively provide technical support to health districts:

“The PHA staff member is reassured about his mastery of the issues he discusses with the DHMT members”. [IDI 9 ].

Proper preparation of technical visits and continuous self-learning also boost self-efficacy:

“Proper preparation for technical support visits increases the chances of satisfaction for PHA staff in their coaching role. This is because good results improve self-efficacy and motivate individuals to move forward”. [DR 13 ]

Motivation was found to be a key mechanism for providing effective technical support to DHMT members. Respondents state that this motivation is extrinsic, i.e. linked to financial incentives and to the working environment and conditions:

“It is clear that the motivation of health workers in the resources-limited context is a delicate issue. However, each PHA office in the country should have minimum funding (considering state wages, risk premiums, and funds from other partners) that needs to be coordinated for adequate technical support to health districts [...]. Boosting the PHA staff's motivation does not solely rely on financial incentives. It also involves enhancing the working environment, teamwork, relationships between colleagues, and the leadership quality of the head of PHA. Recognising and appreciating small achievements and providing opportunities for PHA staff to upgrade their skills can increase the PHA staff's motivation even in a challenging environment”. [IDI 5 ]

Reflexivity was found to be helpful for individual and collective learning. The interviews and the document review showed that PHA staff should be reflexive and instil this attitude in DHMT members.

“The preparatory and debriefing meetings for technical support visits at the PHA office are opportunities for PHA staff to question their practices, share their experiences and learn from each other”. [IDI 9 ] “The PHA staff should develop methods that encourage supported teams to question their performance, practices, working methods regarding the problems that arise, the health district's objectives and their working environment”. [DR 13 ]

Psychological safety was also found to be a key mechanism for individual and collective learning. It depends on the leadership of the head of the PHA office.

“The PHA’s activities are coordinated through regular meetings to discuss necessary actions in response to identified problems. The preparation, debriefing of technical missions and analysis of health district data are suitable opportunities for these discussions. The head of the PHA office should create the right conditions for open and cordial discussions to take place, leading to consensus-based decisions”. [IDI 9 ]

Mechanisms for DHMT members

Perceived credibility of the PHA staff: DHMT members are more likely to accept and actively participate in the technical support process when they perceive their coach as credible. This perception “is nourished by the knowledge, know-how and interpersonal skills of the PHA staff […] and is necessary for him to have a certain leadership, recognition and influence over the DHMT members”. [DR 12 ] This is also explained in the following excerpt:

“The PHA staff improves his or her own knowledge and skills, which enables him or her not to lose face in front of the teams being supported and thus maintain credibility and legitimacy in their eyes”. [DR 13 ]

Perceived relevance of the support: DHMT members are more likely to accept and actively participate in the technical support process when they perceive it as relevant. This positive perception is triggered if “the support is consistent with the legitimate needs of those being supported”. [DR 13 ].

“The management teams in the district are looking for support that genuinely meets their expressed needs. It is essential for the support provided by the PHA staff to align with the actual needs identified at the health district level”. [IDI 5 ]

The scoping review stressed the importance of tailoring capacity building programmes to the needs of district health managers.

“[…] adaptability and flexibility of CBP [capacity building programme] processes make them more responsive as they consider the needs of DHMs [district health managers] and their context, which contribute to increased perceived relevance and sense of ownership by DHMs”. [ScR]

Trust in the PHA staff: supportive relationships foster trust in the PHA staff by DHMT members, which promotes openness and willingness to learn and embrace change:

“Establishing trust is crucial for effective knowledge transfer and district management team members' acceptance of the supervisor’s feedback”. [IDI 9 ] “The supervisor’s attitude will impact the level of trust among the supervised [district management] team members. This will influence their ability to confide in each other and engage in honest and productive dialogue”. [DR 15 ]

The mutual trust between facilitators and participants has been identified in the scoping review as a key driver of participation in the capacity building programme for district health managers.

“[…] supportive interactions between facilitators and DHMs [district health managers], which enable mutual trust and enhance motivation and commitment of DHMs to actively participate in the CBP [capacity building programme] process and to engage with changes”. [ScR]

Self-efficacy: technical support as a capacity-building intervention enhances district management team members’ knowledge and practical skills, which in turn trigger their can-do attitude when carrying out their managerial or leadership tasks at the health district level.

“Capacity building programmes methods, such as team-based training, learning-by-doing approach, a shift from administrative and control to a supporting model of supervision, reflective discussions for continuous learning […] empower DHMs and activate a can-do attitude (self-efficacy)”. [ScR]

Perceived autonomy: effective technical support combined with optimal decision spaces increases the autonomy of DHMT members in performing their duties. The perceived autonomy of DHMTs combined with self-efficacy can increase motivation and improve the team’s performance.

“Effective coaching enhances the autonomy of individuals or teams being coached and continuously improves the team's performance”. [DR 13 ]

Psychological safety was also found to be a key mechanism for learning at the interface between the PHA staff and DHMT members. It depends on the interpersonal skills and posture of PHA staff:

“As you know, technical support is a learning process. It is not a unidirectional but rather a bidirectional process. The DHMT members learn from the PHA staff, and the PHA staff also learn from them. To learn, PHA staff should be humble and know how to communicate and listen effectively. Listening requires giving DHMT members the chance to ask questions and voice their opinions. […] The PHA staff should avoid a hierarchical approach by assuming they are superior and can dictate what DHMT members should do. There needs to be an exchange of ideas; otherwise, the DHMT members will become frustrated and reluctant to share their ideas”. [IDI 2 ]

ICAMO configurations

We identified six ICAMO configurations, two at the provincial level, three at the interface between PHA staff and DHMT members and one at the health district level (Table  3 ).

ICAMO configuration 1

Training in management and facilitation, including relational knowledge and skills (I) targeting PHA staff (A), increases their self-efficacy (M) and motivation (M), leading to improved competencies (O) and commitment to providing technical support to DHMT members (O). A good work climate, promotion of positive values and provision of adequate resources (C) at the PHA office are essential.

Unsupportive leadership in a context of inadequate resources (C) may demotivate (M) PHA staff (A) and lead to the exit of staff, reducing the number of skilled staff at the PHA office and thus jeopardising the technical support process (O).

ICAMO configuration 2

Regular meetings at the PHA office to plan, evaluate and discuss technical support issues (I) offer PHA staff (A) opportunities to share, reflect on and learn from their field experiences, enabling psychological safety (M) among PHA cadres and contributing to reflexivity (M), which leads to improved competencies (O) through individual and collective learning on the condition of safe conversational spaces that values and respects everyone’s opinions and encourages people to speak up (C).

Conversely, a highly hierarchical management culture (C) can create psychological unsafety (M), making PHA staff (A) hesitant to share their opinions for fear of being judged, embarrassed or punished. This inhibits both reflexivity and learning and thus hinders the development of competencies of PHA staff (O).

ICAMO configuration 3

DHMT members’ involvement in identifying their own support needs and planning support visits (I) results in a positive perception of the relevance of the support received (M), encouraging their active participation in the technical support process and improving their competencies (O). This occurs more likely in an environment conducive to learning (i.e. that is judgement-free, fault-accepting, non-threatening and less hierarchical and where there are supportive relationships between PHA staff and DHMT members) (C).

Conversely, vertical supervision visits by disease control programme staff (I) that do not necessarily meet the needs of DHMT members (A), lead to perceptions of irrelevance of such supervision (M), hindering their professional development and ultimately resulting in less than optimal performance (O).

ICAMO configuration 4

DHMT members (A) are likely to participate effectively in technical support and thus improve their competencies (O) if they perceive the PHA staff as credible (M) and trustworthy (M). These positive perceptions of credibility and trustworthiness are triggered if the PHA staff has good management, facilitation and relational skills (A), which allow them to provide effective problem-solving support (I) to DHMT members and set up a conducive learning environment that is judgement-free, fault-accepting, non-threatening and less hierarchical and fosters supportive relationships with DHMT members (C).

However, supervisions by PHA staff members with a hierarchical attitude (I) may be perceived as less credible (M) and trustworthy (M) by the DHMT members (A), hinder their psychological safety (M) and result in weak or reluctant participation in the technical support process (O), ultimately hampering the performance of health districts (O).

ICAMO configuration 5

If competent PHA staff members stimulate meaningful reflections and provide constructive feedback (I), in a learning environment that is judgement-free, fault-accepting, non-threatening and non-hierarchical, and where relationships between PHA staff and DHMT members are supportive (C), then DHMT members may become more reflexive (M), which contributes to individual and collective learning and ultimately improved competencies (O).

ICAMO configuration 6

If supervision (I) increases their competences, DHMT members (A) will be more motivated to develop management initiatives to improve their health districts’ performance (O) because of higher self-efficacy (M) and perceived autonomy (M). Favourable contextual conditions include strong leadership, a supportive work environment with adequate resources and an absence of negative political influences (C).

The initial programme theory

On the basis of the ICAMO configurations, we drafted an initial programme theory:

Adequate training of PHA staff that is based on an action-learning approach and regular meetings to plan, evaluate and discuss technical support for DHMT members improves their competencies by increasing self-efficacy, motivation, psychological safety and reflexivity. Effective leadership, availability of resources and a safe conversational space at the PHA office are crucial context factors.

At the interface between PHA staff and DHMT members, technical support for DHMT members that addresses their needs and provides effective problem-solving, meaningful reflections and constructive feedback can trigger positive perceived relevance of support, positive perceived credibility of PHA staff, trust in PHA staff and psychological safety. This, in turn, can lead to improved competencies of DHMT members through their active participation in technical support processes and individual and collective learning. It requires a conducive environment for learning (judgment-free, accepting faults, non-threatening and non-hierarchical), optimal integration of vertical-specific disease programmes and competent PHA staff with good management, facilitation and relational skills.

At the district level, effective technical support increases the competencies, self-efficacy and perceived autonomy of DHMT members, who will more easily develop initiatives to improve their performance. Favourable contextual conditions include effective leadership, a supportive work environment, adequate resources and an absence of negative political influences.

In summary, the IPT is outlined in Fig.  3

figure 3

In this paper, we identify ICAMO components and formulate six ICAMO configurations explaining how technical support processes bring about expected outcomes by triggering mechanisms for PHA staff and DHMT members under specific contextual conditions.

The IPT emphasises the importance of an action-learning approach at both the provincial and district levels. Action-learning focuses on action (or experience) and reflection as sources of learning. Indeed, intervention components at the provincial level (PHA staff training and meetings) involve action and reflection. Similarly, key features of effective technical support for DHMT members identified in this study (personalised, problem-solving-centred and reflection-stimulating) involve action and reflection. The action-learning approach is rooted in adult learning theories, which are Kolb’s experiential learning theory [ 48 , 49 ], Knowles’ adult learning theory [ 50 ] and Mezirow’s transformative learning theory [ 51 ].

In Kolb’s experiential learning theory, the learning cycle consists of four stages: concrete experience, reflective observation, abstract conceptualisation and active experimentation. This implies that concrete experiences lead to reflective observation, from which abstract concepts are developed and tested in new experiences [ 48 , 49 ]. When applied to the technical support process, PHA staff and DHMT members can learn from their practical experiences by reflecting on them. Knowles’ adult learning theory highlights the importance of self-directed learning, experiences (including errors), perceived relevance, problem-solving and intrinsic motivation in the learning process [ 50 ]. By involving DHMT members in the planning and evaluation of their own learning experiences and tailoring technical support approaches to their unique needs and experiences, a learner-centred environment can be created. Mezirow’s transformative learning theory suggests that learning occurs when people critically reflect on their values, beliefs and assumptions, leading to new and meaningful perspectives. This theory emphasises six key elements: individual experience, critical reflection, dialogue, holistic orientation, awareness of context and authentic relationships [ 51 ]. PHA can facilitate transformative discussions that help DHMT members question and reframe their perspectives. From the preceding, it is evident that the three adult learning theories encompass key technical support features (personalised, problem-solving-centred and reflection-stimulating support). Integrating these adult learning theories within the technical support process may enhance its effectiveness by promoting experiential, transformative and self-directed learning tailored to the unique context of DHMT members. Additionally, these theories highlight the importance of perceived relevance in the learning process.

This perceived relevance can be linked to the health belief model [ 52 ] and to the integrated theory of health behaviour change [ 53 ]. The health belief model suggests that the perceived benefits or positive consequences of a health behaviour can influence its adoption [ 52 ]. According to the integrated theory of health behaviour change, enhancing personal perceptions (such as self-efficacy, outcomes expectancy and goal congruence) and social support (emotional, instrumental or informational) can lead to engagement in a health behaviour [ 53 ]. Similarly, DHMT members are more likely to accept and actively participate in the technical support process if they perceive it as relevant to their needs and expectations (outcomes expectancy and goal congruence). This perception of relevance is a crucial mechanism in their engagement because they may believe that the technical support will equip them with new knowledge and skills (benefits) they can use to enhance their professional growth. When technical support is aligned with their professional needs and challenges, it becomes more effective and meaningful.

Effective technical support involves supportive relationships between PHA staff and DHMT members and can enhance the motivation, self-efficacy and autonomy of DHMT members as well as PHA staff. These mechanisms refer to self-determination theory [ 40 ], according to which every person seeks to fulfil three fundamental psychological needs – autonomy, competence and relatedness – which are essential for their optimal motivation, engagement and well-being [ 40 , 42 ]. Competence involves feeling effective or having self-efficacy when performing work tasks. Bandura [ 37 ] proposed four sources of self-efficacy: mastery experiences, vicarious experiences, verbal persuasion and physiological and affective states. During the technical support process, PHA staff can boost the self-efficacy of DHMT members by leveraging these four sources. First, they can encourage the implementation of sound management practices and skill development to give DHMT members positive mastery experiences. Second, they can act as role models or share success stories from other DHMTs to enhance the belief of DHMT members that they too can perform effectively in their roles (vicarious experiences). Third, PHA staff can provide positive feedback and express confidence in the capabilities of DHMT members, thereby increasing their self-efficacy through social persuasion. Finally, a supportive and positive emotional environment can be fostered by the PHA staff during the technical support process in order to reduce stress and enhance overall well-being, leading to a heightened sense of self-efficacy among DHMT members (affective states). By systematically reinforcing each of these four sources of self-efficacy, PHA staff can create a learning environment that empowers DHMT members and builds and sustains their confidence in their abilities. This learning environment can foster relatedness or a sense of belonging to a social group [ 40 , 42 , 54 ], which can lead to psychological safety [ 55 ]. Enhanced self-efficacy can lead to autonomy, allowing DHMT members to have freedom and control over their actions.

Psychological safety was found to be a mediator between antecedents such as supportive leadership behaviour, supportive organisational practices and relationship networks and positive work outcomes such as team learning, performance, innovation [ 45 , 56 ] and reflexivity [ 46 ]. Team learning – known as a continuous process of questioning, reflecting, experimenting, seeking feedback and discussing outcomes or errors [ 43 ] – is linked to reflexivity. The level of reflexivity within a team is influenced by various factors, such as the level of trust and psychological safety among team members, a shared vision, diversity and leadership style. Higher levels of reflexivity can lead to increased innovation, effectiveness, and creativity within a team [ 46 , 56 , 57 , 58 ]. Psychological safety is related to trust, another key mechanism for the learning process. Both psychological safety and trust refer to the climate within a team regarding the expectation of cooperative or non-harming behaviour of the PHA staff or other DHMT members [ 46 , 59 , 60 , 61 ]. Trust is essential in daily workplace dynamics and fosters positive relationships. It has been linked to the improved intrinsic motivation of health workers [ 62 ], improved team performance and positive work outcomes, such as better organisational citizenship behaviour [ 63 ]. The relational skills of PHA staff are incredibly important for activating these mechanisms. In fact, PHA staff must adopt a suitable attitude that promotes reflection, ensures psychological safety and instils trust to facilitate an effective learning process. Fostering psychological safety within a team creates an environment where team members feel comfortable expressing themselves, taking risks, and learning from experiences. This, in turn, enhances reflexivity – individual and collective reflection – which may in turn lead to increased innovation and effectiveness as the team continually adapts, learns and generates creative solutions to challenges.

Our findings indicate that technical support is a multi-level process operating at the PHA office, the PHA staff-DHMT members interface and the health district levels. Figure  2 illustrates the connection between these levels, showing the influence of the higher level’s outcomes on the immediate lower level’s context. This phenomenon is referred to as the ‘ripple effect’ in realist literature [ 64 ]. It is a consequence of complex interdependencies within the health system, requiring better coordination of activities across levels to ensure that any change in one level does not negatively affect other levels [ 65 ].

Rigour and trustworthiness

In the realist approach, the rigour of a study depends on ‘the trustworthiness of the evidence source and the coherence of programme theory’ [ 66 ]. To ensure the trustworthiness of our study findings, we gathered data from primary and secondary sources (interviews with programme designers, scoping review and document review) and triangulated them during data analysis. Triangulation involves comparing and contrasting information from multiple perspectives to gain a more comprehensive understanding of the research phenomenon [ 67 ]. By cross-verifying data from different sources and methods, triangulation helped us to minimise the bias associated with a single method or data source, and this increased the reliability, credibility and validity of our findings. Our analysis went beyond the mere thematic categorisation of intervention features, context, actors, mechanisms and outcomes (ICAMO components) to delve deeper into the relationships between them (i.e. developing ICAMO configurations) [ 68 ].

We enhanced the coherence of our programme theory in various ways. First, we searched for rival theories, i.e. alternative statements hypothesising how the same programme resources could result in different responses and outcomes [ 69 ]. Second, we discussed the articulation of our programme theory with relevant substantive theories, such as adult learning theories (including Kolb’s experiential learning theory, Knowles’ adult learning theory and Mezirow’s transformative learning theory), health behaviour change theories (such as the health belief model and the integrated theory of health behaviour change) and self-determination theory [ 66 , 68 ]. These theories complement each other in explaining the learning process. Indeed, most adult learning theories underscore the significance of self-directed learning and the real-life application of knowledge. They are aligned with the principles of self-determination theory that emphasise autonomy in the learning process. Health behaviour change theories offer insights into the factors influencing DHMT members’ behaviour and motivation. Integrating these theories with adult learning principles may enable PHA staff to tailor technical support to address the specific needs, challenges and motivations of DHMT members. Self-determination theory, according to which intrinsic motivation is the result of fulfilling the psychological needs of autonomy, competence and relatedness, can be combined with adult learning and health behaviour theories. However, the integration of multiple theories is a complex process, requiring a nuanced understanding of each theory’s principles. To facilitate this integration, it is essential to clearly define the connections between theories and provide practical guidelines for PHA staff to apply them cohesively in supporting DHMT members.

Finally, we discussed and sought feedback from supervisors who are experts in realist evaluation and health systems and policy research and are well-versed in the Congolese health system. Such discussions and feedback ensured the appropriateness of methods, data analysis and interpretations and contributed to the credibility of our study. Furthermore, we adhered to the reporting standards for realist evaluation (RAMESES II checklist, Additional file 3 ) [ 70 ] and qualitative research (COREQ checklist, Additional file 4 ) [ 71 ]. These standards are designed to improve the comprehensiveness, consistency and rigour of research reporting. In addition, they enhance transparency, facilitate reproducibility and enable quality assessment of studies. They help readers to better understand the design, conduct, analysis and findings of the studies [ 70 , 71 , 72 , 73 ]. However, reporting standards have some limitations. One of the limitations is that they primarily address the reporting phase of research, emphasizing how research is communicated rather than how it is conducted. While transparent reporting is crucial, it does not guarantee that there are no methodological flaws or biases in the actual research process [ 72 ]. To minimize this limitation, we combined the use of these checklists with the quality standards for realist evaluation for evaluators and peer-reviewers set by the RAMESES II Project [ 74 ].

Strengths and limitations of the study

A key strength of this study is the use of multiple strategies (data source triangulation, search for rival theories, articulation with relevant substantive theories, expert audit and adherence to reporting standards) to ensure the rigour (trustworthiness and coherence) of our study.

However, this study has some limitations. First, the insights from the scoping review were helpful in developing the IPT. However, it is important to note that only a few studies were conducted in fragile settings like the DRC. This raises questions about the necessary contextual factors that are needed for effective capacity development processes. Second, we presented the ICAMO configurations discretely and linearly. In reality, we expect the underlying pathways to be more intricate and interdependent. Technical support for DHMT members is, indeed, a complex capacity-building intervention that a configurational analysis cannot fully describe. Pawson and Tilley advise realist researchers to acknowledge the complexity of their subject matter and remain humble in their approach, recognising that their ‘understanding will always be partial and provisional’ [ 20 ]. Realist inquiry outputs are approximate in nature, which requires them to accumulate over time through recurrent theory testing and refinement cycles [ 26 , 28 ]. Third, it is worth noting the possibility of recall bias, particularly for the designers of the intervention which is over 5 years old, and the social desirability bias during this study. We took measures to minimize these biases by using triangulation of data sources and methods [ 30 ].

Technical support for DHMTs is a complex intervention in a complex health system. This study allowed us to identify six ICAMO configurations that explain how effective technical support, which is personalised, problem-solving-centred and reflection-stimulating, can improve the competencies of DHMT members. This improvement can be achieved through the activation of various mechanisms, such as the positive perceived relevance of the support, positive perceived credibility of PHA staff, trust in PHA staff, psychological safety, reflexivity, self-efficacy and perceived autonomy. These mechanisms operate within specific contextual conditions, including an enabling learning environment, the integration of vertical programs, competent PHA staff, optimal decision-making space, supportive work conditions, availability of resources and the absence of negative political influences. These ICAMO configurations will be tested in subsequent empirical studies.

Availability of data and materials

The dataset supporting the conclusions of this article is included in the article (and its supplemental files).

Abbreviations

  • Democratic Republic of Congo

District health management team

Health system strengthening strategy

Intervention–context–actor–mechanism–outcomes

Initial programme theory

Ministry of Health

Provincial Health Administration

  • Realist evaluation

Meessen B, Van Damme W. Systèmes de santé des pays à faible revenu: Vers une révision des configurations institutionnelles? Mondes Dev. 2005;33(3):59–73.

Google Scholar  

Alliance for Health Policy and Systems Research. Strengthening health systems: the role and promise of policy and systems research. Geneva; 2004.

Papanicolas I, Rajan D, Karanikolos M, Soucat A, Soucat A, editors. Health system performance assessment. A framework for policy analysis. Geneva: World Health Organization; 2022.

Senkubuge F, Modisenyane M, Bishaw T. Strengthening health systems by health sector reforms. Glob Health Action. 2014;7(SUPP1):23568.

Article   PubMed   Google Scholar  

Bossert T. Analyzing the decentralization of health systems in developing countries: Decision space, innovation and performance. Soc Sci Med. 1998;47(10):1513–27.

Article   CAS   PubMed   Google Scholar  

World Health Organization. The World health report 2000: health systems: improving performance [Internet]. Geneva: World Health Organization; 2000. https://apps.who.int/iris/handle/10665/42281

World Health Organization. Report on the performance of health systems in the WHO Africa Region. 2020.

Fullman N, Yearwood J, Abay SM, Abbafati C, Abd-Allah F, Abdela J, et al. Measuring performance on the Healthcare Access and Quality Index for 195 countries and territories and selected subnational locations: a systematic analysis from the Global Burden of Disease Study 2016. Lancet. 2018;391(10136):2236–71.

Article   Google Scholar  

Kalambay H, Wim N, Lerberghe V. Improving health system efficiency: Democratic Republic of the Congo: improving aid coordination in the health sector. 2015. https://apps.who.int/iris/handle/10665/186673 . Accessed 17 May 2022.

Ministère de la Santé. Stratégie de Renforcement du Système de Santé. République Démocratique du Congo; 2010.

Ministère de la Santé. Stratégie de Renforcement du Système de Santé. République Démocratique du Congo; 2006.

Bosongo SI, Mukalenge FC, Tambwe AM, Criel B. Les médecins prestataires à la première ligne des soins dans la ville de Kisangani en République Démocratique du Congo: vers une typologie. Afr J Prim Health Care Fam Med. 2021;13(1):1–8.

Présidence de la République. Constitution de la République Démocratique Du Congo. Journal Officiel de la République Démocratique du Congo. 2006;

Kahindo M, Schirvel C, Godelet E, Wodon A, Porignon D, Bonami M. Réforme des structures intermédiaires de santé en République démocratique du Congo. Sante Publique. 2014;26(6):10.

Ministère de la Santé Publique RDC. Référentiel de compétences intégré de l’équipe cadre de la zone de santé. Pour les situations professionnelles relatives au management de la zone de santé en RDC. Kinshasa; 2010.

Kahindo MJB, Mitangala NP, Tsongo ME, Mahamba N, Kahandukya NL, Kubuya Bonane J, et al. Soutien du niveau intermédiaire du système au district de santé: Perceptions des équipes de district de santé du Nord Kivu à l’est de la RDC. Int J Innov Appl Stud. 2021;33(2):258–71.

Chuy KD, Criel B, Belrhiti Z, Mwembo TA, Chenge MF. Soutien du niveau intermédiaire du système de santé aux équipes cadres des districts sanitaires: Le cas de la ville de Lubumbashi, République Démocratique du Congo. Int J Adv Res (Indore). 2020;8(3):1114–39.

Prashanth NS, Marchal B, Devadasan N, Kegels G, Criel B. Advancing the application of systems thinking in health: a realist evaluation of a capacity building programme for district managers in Tumkur, India. Health Res Policy Syst. 2014;12(1):42. https://doi.org/10.1186/1478-4505-12-42 .

Article   PubMed   PubMed Central   Google Scholar  

Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.

Pawson R, Tilley N, Pawson R, Tilley N. Realist Evaluation Realist Evaluation. 2004; http://www.communitymatters.com.au/RE_chapter.pdf

Dalkin SM, Greenhalgh J, Jones D, Cunningham B, Lhussier M. What’s in a mechanism? Development of a key concept in realist evaluation. Implement Sci. 2015;10(1):1–7.

Lacouture A, Breton E, Guichard A, Ridde V. The concept of mechanism from a realist approach: a scoping review to facilitate its operationalization in public health program evaluation. Implement Sci. 2015;10(1):1–10.

Lemire S, Kwako A, Nielsen SB, Christie CA, Donaldson SI, Leeuw FL. What is this thing called a mechanism? Findings from a review of realist evaluations. In: Causal Mechanism in Evaluation. New Direct. 2020.

Smeets RGM, Hertroijs DFL, Mukumbang FC, Kroese MEQL, Ruwaard D, Elissen AMJ. First things first: how to elicit the initial program theory for a realist evaluation of complex integrated care programs. Milbank Q. 2022;100(1):151–89.

Mukumbang FC, Kabongo EM, Eastwood JG. Examining the application of retroductive theorizing in realist-informed studies. Int J Qual Methods. 2021;20:16094069211053516.

Jagosh J. Retroductive theorizing in Pawson and Tilley’s applied scientific realism. J Crit Realism. 2020;19(2):121–30.

Merton RK. Social Theory and Social Structure. Enlarged. New York: The Free Press; 1968.

Marchal B, van Belle S, van Olmen J, Hoerée T, Kegels G. Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation. 2012;18(2):192–212.

Marchal B, Kegels G, Van Belle S. Theory and realist methods. In: Emmel N, Greenhalgh J, Manzano A, Monaghan M, Dalkin S, editors. Doing realist research. 23th edition. 2017.

Mukumbang FC. A realist evaluation of the antiretroviral treatment adherence club programme in the metropolitan area of the Western Cape Province, South Africa. University of the Western Cape; 2018. http://etd.uwc.ac.za

Bosongo S, Belrhiti Z, Ekofo J, Kabanga C, Chenge F, Criel B, et al. How capacity building of district health managers has been designed, delivered and evaluated in sub-Saharan Africa: a scoping review and best fit framework analysis. BMJ Open. 2023;13(8): e071344. https://doi.org/10.1136/bmjopen-2022-071344 .

Mukumbang FC, Van Belle S, Marchal B, Van Wyk B. Towards developing an initial programme theory: Programme designers and managers assumptions on the antiretroviral treatment adherence club programme in primary health care facilities in the metropolitan area of western cape province, South Africa. PLoS ONE. 2016;11(8): e0161790.

Bowen GA. Document analysis as a qualitative research method. Qual Res J. 2009;9(2):27–40.

Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Burgess B, editors. Analyzing qualitative data. 1st ed. London: Routledge; 1994.

Kleinheksel AJ, Rockich-Winston N, Tawfik H, Wyatt TR. Qualitative research in pharmacy education. Demystifying content analysis. Am J Pharm Educ. 2020;84(1):127–37.

Mukumbang FC, Marchal B, Van Belle S, van Wyk B. Using the realist interview approach to maintain theoretical awareness in realist studies. Qual Res. 2020;20(4):485–515.

Bandura A. Self-efficacy: the exercise of control. New York: W H Freeman; 1997.

Bandura A. Social foundations of thought and action: a social cognitive theory. Hoboken: Prentice-Hall, Inc.; 1986.

Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215.

Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation. Soc Dev Well-Being American Psychol. 2000;55(1):68–78.

CAS   Google Scholar  

Deci EL, Ryan RM. Self-Determination Theory. In: International Encyclopedia of the Social & Behavioral Sciences: Second Edition. Elsevier Inc.; 2015. p. 486–91.

Legault L. Self-Determination Theory. In: Encyclopedia of Personality and Individual Differences. Cham: Springer International Publishing; 2017. p. 1–9. https://doi.org/10.1007/978-3-319-28099-8_1162-1

Edmondson A. Psychological safety and learning behavior in teams. Adm Sci Q. 1999;44:350–83.

Edmondson AC, Lei Z. Psychological safety: the history, renaissance, and future of an interpersonal construct. Annu Rev Organ Psych Organ Behav. 2014;1(1):23–43.

West MA. View graph of relations Reflexivity, revolution, and innovation in work teams. Beyerlein M, Johnson D, Beyerlein S, editors. Stamford: JAI Press; 2000. 1–29 p.

Widmer PS, Schippers MC, West MA. Recent developments in reflexivity research: a review. Psychol Everyday Activity. 2009;2(2):2–11.

Rousseau DM, Sitkin SB, Burt RS, Camerer C. Not so different after all: a cross-discipline view of trust. Acad Manag Rev. 1998;23:393–404.

Kolb DA, Boyatzis RE, Mainemelis C, Sternberg RJ, Zhang LF. Experiential learning theory: previous research and New Directions. 2000.

Kolb AY, Kolb DA. Experiential learning theory: A dynamic, holistic approach to management learning, education and development. In: The SAGE Handbook of Management Learning, Education and Development. SAGE Publications Inc.; 2009. p. 42–68.

Knowles MS, Holton EF, Swanson RA. The adult learner: the definitive classic in adult education and human resource development. 6th ed. Amsterdam: Elsevier; 2005.

Book   Google Scholar  

Mezirow J. Transformative dimensions of adult learning. 1st ed. San Francisco: Jossey-Bass; 1991.

Green EC, Murphy EM, Gryboski K. The Health Belief Model. In: The Wiley Encyclopedia of Health Psychology. 2020, 211–4. https://doi.org/10.1002/9781119057840.ch68

Ryan P. Integrated theory of health behavior change: background and intervention development. Clin Nurse Spec. 2009;23:161–70 ( quiz 171 ).

Belrhiti Z, Van Damme W, Belalia A, Marchal B. Unravelling the role of leadership in motivation of health workers in a Moroccan public hospital: a realist evaluation. BMJ Open. 2020;10(1):1–17.

Aboud K, Xiongying N, Qasim M. Impact of ethical leadership on employees’ psychological safety and voice behavior; with mediating role of belongingness. Int J Sci Bus. 2023;20(1):42–57.

Carter SM, West MA. Reflexivity, effectiveness and mental health in BBC-TV production teams. Small Group Res. 1998;29(5):583–601.

Schippers MC, West MA, Edmondson AC. Team Reflexivity and Innovation. In: The Wiley Blackwell Handbook of the Psychology of Team Working and Collaborative Processes. Wiley; 2017. p. 459–78.

Tjosvold D, Tang MML, West M. Reflexivity for team innovation in China: the contribution of goal interdependence. Group Organ Manag. 2004;29(5):540–59.

Gilson L. Trust in health care: theoretical perspectives and research needs. J Heal Organ Manag. 2006;20(5):359–75.

Gilson L. Trust and the development of health care as a social institution. Soc Sci Med. 2003;56:1453.

Goudge J, Gilson L. How can trust be investigated? Drawing lessons from past experience. Soc Sci Med. 2005;61:1439–51.

Okello DRO, Gilson L. Exploring the influence of trust relationships on motivation in the health sector: a systematic review. Hum Resour Health. 2015;13(1):16.

Simpson JA, Vieth G. Trust and psychology: psychological theories and principles underlying interpersonal trust. In: Krueger F, editor. The neurobiology of trust. Cambridge: Cambridge University Press; 2021. p. 15–35.

Chapter   Google Scholar  

Jagosh J, Bush PL, Salsberg J, Macaulay AC, Greenhalgh T, Wong G, et al. A realist evaluation of community-based participatory research: partnership synergy, trust building and related ripple effects. BMC Public Health. 2015;15(1):725. https://doi.org/10.1186/s12889-015-1949-1 .

Trickett EJ, Beehler S. The ecology of multilevel interventions to reduce social inequalities in health. Am Behav Sci. 2013;57(8):1227–46.

Dada S, Dalkin S, Gilmore B, Hunter R, Mukumbang FC. Applying and reporting relevance, richness and rigour in realist evidence appraisals: advancing key concepts in realist reviews. Res Synth Methods. 2023;14(3):504–14.

Patton MQ. Qualitative evaluation and research methods. 3rd ed. Thousand Oaks: Sage Publications; 2002.

The RAMESES II Project. Quality standards for realist evaluation: for evaluators and peer-reviewers. 2017. www.ramesesproject.org . Accessed 8 Sep 2023.

Jagosh J, Stott H, Halls S, Thomas R, Liddiard C, Cupples M, et al. Benefits of realist evaluation for rapidly changing health service delivery. BMJ Open. 2022;12(7): e060347.

Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med. 2016;14(1):96.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. https://doi.org/10.1093/intqhc/mzm042 .

Wharton T. Rigor, transparency, and reporting social science research: why guidelines don’t have to kill your story. Res Soc Work Pract. 2017;27(4):487–93.

McLeroy KR, Garney W, Mayo-Wilson E, Grant S. Scientific reporting: raising the standards. Health Educ Behav. 2016;43(5):501–8.

The RAMESES II Project. Quality standards for realist evaluation. For evaluators and peer-reviewers. 2017. www.ramesesproject.org

Download references

Acknowledgements

We thank the study’s participants for their insightful information.

This work was supported by the Directorate-General Development Cooperation and Humanitarian Aid, Belgium, in collaboration with the Institute of Tropical Medicine, Antwerp as a part of the doctoral programme of SB, grant number 911063/70/130. The funder had no role in the whole process of the review, from the design to the publication.

Author information

Authors and affiliations.

Faculté de Médecine et Pharmacie, Université de Kisangani, Kisangani, Democratic Republic of Congo

Samuel Bosongo & Faustin Chenge

École de Santé Publique, Université Libre de Bruxelles, Brussels, Belgium

Samuel Bosongo & Yves Coppieters

Institute of Tropical Medicine, Antwerp, Belgium

Samuel Bosongo, Bart Criel & Bruno Marchal

Centre de Connaissances en Santé en République Démocratique du Congo, Kinshasa, Democratic Republic of Congo

Mohammed VI University of Sciences and Health (UM6SS), Casablanca, Morocco

Zakaria Belrhiti

Centre Mohammed VI de la Recherche et Innovation (CM6), Rabat, Morocco

Ecole de Santé Publique, Faculté de Médecine, Université de Lubumbashi, Lubumbashi, Democratic Republic of Congo

Faustin Chenge

You can also search for this author in PubMed   Google Scholar

Contributions

S.B., Z.B. and B.M. conceptualised the study. S.B. collected, analysed data and drafted the initial manuscript. S.B., Z.B., B.M., F.C., B.C. and Y.C. contributed to the manuscript revision. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Samuel Bosongo .

Ethics declarations

Ethics approval and consent to participate.

The protocol was approved by the Institutional Review Board of the Institute of Tropical Medicine, Antwerp (reference IRB no. 1654/22) and the Medical Ethics Committee of the University of Lubumbashi (Reference no. UNILU/CEM/005/2023).

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Description of documents included in the review document.

Additional file 2.

Interview guide for programme designers.

Additional file 3.

Checklist for realist evaluation studies.

Additional file 4.

COREQ checklist.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Bosongo, S., Belrhiti, Z., Chenge, F. et al. The role of provincial health administration in supporting district health management teams in the Democratic Republic of Congo: eliciting an initial programme theory of a realist evaluation. Health Res Policy Sys 22 , 29 (2024). https://doi.org/10.1186/s12961-024-01115-9

Download citation

Received : 05 October 2023

Accepted : 31 January 2024

Published : 20 February 2024

DOI : https://doi.org/10.1186/s12961-024-01115-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Technical support
  • District health management teams
  • Provincial health administration

Health Research Policy and Systems

ISSN: 1478-4505

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

problem solving for evaluation

Help | Advanced Search

Computer Science > Artificial Intelligence

Title: geoeval: benchmark for evaluating llms and multi-modal models on geometry problem-solving.

Abstract: Recent advancements in Large Language Models (LLMs) and Multi-Modal Models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2000 problems, a 750 problem subset focusing on backward reasoning, an augmented subset of 2000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs on solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67\% accuracy rate on the main subset but only a 6.00\% accuracy on the challenging subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.

Submission history

Access paper:.

  • Download PDF
  • Other Formats

license icon

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

404 Not found

Subscribe to the PwC Newsletter

Join the community, edit social preview.

problem solving for evaluation

Add a new code entry for this paper

Remove a code repository from this paper, mark the official implementation from paper authors, add a new evaluation result row.

  • GEOMETRY PROBLEM SOLVING

Remove a task

Add a method, remove a method, edit datasets, geoeval: benchmark for evaluating llms and multi-modal models on geometry problem-solving.

15 Feb 2024  ·  Jiaxin Zhang , Zhongzhi Li , Mingliang Zhang , Fei Yin , ChengLin Liu , Yashar Moshfeghi · Edit social preview

Recent advancements in Large Language Models (LLMs) and Multi-Modal Models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2000 problems, a 750 problem subset focusing on backward reasoning, an augmented subset of 2000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs on solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67\% accuracy rate on the main subset but only a 6.00\% accuracy on the challenging subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.

Code Edit Add Remove Mark official

Tasks edit add remove, datasets edit.

problem solving for evaluation

Results from the Paper Edit Add Remove

Methods edit add remove.

  • Open access
  • Published: 17 February 2024

The effectiveness of simple heuristic features in sensor orientation and placement problems in human activity recognition using a single smartphone accelerometer

  • Arnab Barua 1 ,
  • Xianta Jiang 1 &
  • Daniel Fuller 2  

BioMedical Engineering OnLine volume  23 , Article number:  21 ( 2024 ) Cite this article

104 Accesses

Metrics details

Human activity Recognition (HAR) using smartphone sensors suffers from two major problems: sensor orientation and placement. Sensor orientation and sensor placement problems refer to the variation in sensor signal for a particular activity due to sensors’ altering orientation and placement. Extracting orientation and position invariant features from raw sensor signals is a simple solution for tackling these problems. Using few heuristic features rather than numerous time-domain and frequency-domain features offers more simplicity in this approach. The heuristic features are features which have very minimal effects of sensor orientation and placement. In this study, we evaluated the effectiveness of four simple heuristic features in solving the sensor orientation and placement problems using a 1D-CNN–LSTM model for a data set consisting of over 12 million samples.

We accumulated data from 42 participants for six common daily activities: Lying, Sitting, Walking, and Running at 3-Metabolic Equivalent of Tasks (METs), 5-METs and 7-METs from a single accelerometer sensor of a smartphone. We conducted our study for three smartphone positions: Pocket, Backpack and Hand. We extracted simple heuristic features from the accelerometer data and used them to train and test a 1D-CNN–LSTM model to evaluate their effectiveness in solving sensor orientation and placement problems.

We performed intra-position and inter-position evaluations. In intra-position evaluation, we trained and tested the model using data from the same smartphone position, whereas, in inter-position evaluation, the training and test data was from different smartphone positions. For intra-position evaluation, we acquired 70–73% accuracy; for inter-position cases, the accuracies ranged between 59 and 69%. Moreover, we performed participant-specific and activity-specific analyses.

Conclusions

We found that the simple heuristic features are considerably effective in solving orientation problems. With further development, such as fusing the heuristic features with other methods that eliminate placement issues, we can also achieve a better result than the outcome we achieved using the heuristic features for the sensor placement problem. In addition, we found the heuristic features to be more effective in recognizing high-intensity activities.

Human activity recognition (HAR) is the process of enabling computers to recognize human activities by analyzing patterns in different data types, including sensor data, images, and videos. Research on HAR is important as it is the principal method for accomplishing applications, such as identifying risk factors regarding depression [ 1 ], diabetes [ 2 ], health condition surveillance [ 3 , 4 ], eldercare [ 5 ], sports performance analysis [ 6 ], and abnormal activity identification [ 7 ]. Since HAR is the primary foundation for the successful implementation of many applications, researchers are trying to overcome the challenges which cause inaccuracy in HAR. Sensor data is one of the most reliable and popular data types used in HAR. Sensor data includes data from accelerometers, gyroscopes, and magnetometers [ 8 , 9 , 10 ]. Studies have used these sensors in divergent ways to accumulate data for HAR. Some researchers attached the sensors separately to different body parts [ 11 , 12 , 13 ], and some used sensors embedded in smartphones [ 14 , 15 , 16 , 17 , 18 ] or smartwatches [ 19 , 20 , 21 ]. Among these different types of sensory devices and placements, smartphones are efficient, feasible and beneficial to HAR research, because they address a number of advantages, including applicability to a large population. Almost every smartphone contains an accelerometer and a gyroscope sensor. Data from both of these sensors are capable of distinguishing different human activities, which means they are feasible for HAR applications. Smartphones are also an inseparable part of daily human life. As a result, researchers emphasized improving the HAR using smartphone sensors by adapting various techniques to diminish the difficulties posed by smartphones in HAR.

Advancing the HAR process using smartphone sensors requires the researchers to overcome some significant challenges related to sensor orientation, sensor placement, and algorithm choice. The sensor orientation problem is one of the most concerning problems faced when using smartphones in HAR, as a smartphone can be kept in different orientations, as depicted in Fig.  1 . A user can keep the smartphone in any orientation and perform different activities. When two different users perform the same activity while keeping the smartphone in different orientations, the sensor data becomes different, making it hard for HAR methods to identify the sensor data as the same activity. Many studies have proposed different methods to deal with the sensor orientation problem of the smartphone in HAR. Researchers also had to propose various approaches to diminish the sensor placement problems [ 22 , 23 ]. The sensor placement problem happens as smartphone users tend to keep their smartphones in different body locations, including backpacks, hands, or pockets. The smartphone sensors, particularly the accelerometer and gyroscope sensors, generate non-identical patterns for similar activity if the smartphone is kept in non-identical locations. For dealing with the sensor orientation and placement problem, researchers generally try to extract features with no orientation or placement effect that could generate substantially different sensor patterns for different activities. For example, [ 24 ] used extracted features in their proposed activity recognition process, where they included data from four different body locations (coat pocket, hand, trouser pocket and bag) and for five human activities (going upstairs, going downstairs, walking, standing and running). They started by extracting horizontal and vertical acceleration data from a raw accelerometer to diminish the influence of device orientation. Later, they extracted eight features from the raw gyroscope signal and separated horizontal and vertical accelerations to develop a position identification system. Finally, they performed feature selection, and using this position recognition system, they conducted some data adjustments to the selected features, which were later used in their activity recognition process. They achieved an accuracy of 91.27% using a Support Vector Machine (SVM) with a 4-fold cross-validation technique. Chen and Shen [ 25 ] extracted 89 time and frequency domain features from smartphones’ accelerometers and gyroscope sensors to make the activity recognition process orientation invariant and position independent. They then performed feature selection and feature normalization on the extracted features. Using these features, they evaluated the performance of three classifiers, K-Nearest Neighbours (KNN), Random Forest (RF), and SVM in recognition of five human activities (descending stairs, ascending stairs, walking, jogging and jumping) for five non-identical smartphone locations (right upper arm, right hand, right jacket pocket, right trousers pocket and waist). They considered different validation procedures named one-to-one, all-to-one and rest-to-one and compared the performance of the classifiers for different validation procedures. Yurtman and Barshan [ 26 ] extracted 9 heuristic features from the data of different sensors available in five public data sets, which they claimed to be free of the influence of sensor orientation. They evaluated the performance of these 9 features in HAR using four machine learning algorithms and found them compelling enough to diminish the orientational effects. Along with these studies, many other studies extracted features to solve the sensor orientation and location dependency problem [ 27 , 28 , 29 , 30 ]. However, along with feature extraction process, coordinate transformation is a promising method to address the sensor orientation and location dependency problem.

figure 1

Possible orientations for a smartphone in a particular placement

In the coordinate transformation approach, first, a global reference coordinate system is discovered, and then all the signals are projected to that reference system. Guo et al. [ 31 ] used a coordination transformation approach on the gyroscope signal from a smartphone fused with a motif discovery algorithm to find activity patterns and then developed a Vector Space Model for classification purposes. They used their approach on a data set containing smartphone signals from four different body positions (left upper arm, the shirt pocket, the trousers front pocket, and the behind trouser pocket) and four different orientations and performed cross-orientation and cross-placement validation. Chen et al. [ 32 ] also performed coordination transformation by calculating quaternion to transform the linear acceleration signal from the device-coordinate system to the earth-coordinate system. Following, they extracted the first two principal components from the transformed acceleration signal to eliminate the direction effect for different activities. In addition, they extracted time and frequency domain features to make their approach more reliable and accurate. To validate their method, they collected data from a smartphone placed in three different positions (pants’ pocket, shirt’s pocket and backpack) and three different orientations. They performed a leave-one-orientation-out cross-validation technique using an Online SVM algorithm and compared results for different orientations, placements and participants. Ustev et al. [ 33 ] also performed coordinate transformation and feature extraction using an accelerometer, gyroscope, and magnetic sensor to eliminate the orientational effect of the smartphone sensor. They evaluated their method for two smartphone orientations (vertical and horizontal) placed in trouser pockets. They achieved 97% accuracy in recognizing five human activities using a KNN classifier. A summary of the studies discussed above is presented in Table  1 . In brief, several studies and methods have been used to address the sensor orientation and placement problem. However, the classification algorithm also plays an important role in HAR classification accuracy, in particular deep learning algorithms are potentially promising given a sufficiently large data set.

Studies have evaluated the performance of divergent machine learning algorithms in HAR and provided comparisons to decide the most suitable classifiers to use. Early classification studies used simple machine learning classifiers, such as SVM [ 27 , 34 , 35 , 36 ], RF [ 37 , 38 , 39 ], KNN [ 40 , 41 ], and decision trees [ 42 , 43 ]. These were employed because of their low complexity and resource-efficient nature. However, the advancement of computational resources enabled the usage of deep learning algorithms, such as artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), long–short-term memory (LSTM), and gaited recurrent units (GRU). These deep learning algorithms offer additional advantages in HAR, especially CNN and LSTM because of CNN’s automated feature extraction capability and LSTM’s ability to persist older information from time series data. Yang et al. [ 44 ] used CNN’s ability of automatic feature learning and found it to outperform four conventional machine learning algorithms in recognizing 18 human activities and 12 hand gestures. They also concluded that CNN was suitable for online HAR. Zeng et al. [ 45 ] also exploited the feature extraction ability of CNN for HAR on three public data sets (Opportunity, Skoda and Actitracker) and acquired an accuracy of 88.19%, 76.83%, and 96.88% on Skoda, Opportunity, and Antitracker, respectively, using the CNN-based model. Xu and Qiu [ 46 ] evaluated the feature extraction capability of CNN in recognizing six daily human activities (sitting, standing, walking, jogging, upstairs, and downstairs) using accelerometer signals. They achieved an accuracy of 94.2%, which outperformed traditional machine learning algorithms, such as Decision Trees (J48) and SVM. There are also other studies that used CNN as their final classification model, along with their early data pre-processing layer, to enhance the recognition rate of human activities [ 47 , 48 , 49 ]. Along with CNN, another deep neural network variation called RNN is being widely used in HAR [ 50 , 51 , 52 ]. RNN has few variations of itself, and among them, LSTM is useful in HAR, especially when studies combine the information-persistence ability of LSTM with the feature extraction capability of CNN. Xia et al. [ 53 ] utilized the combination of CNN and LSTM, also called CNN–LSTM, to evaluate its performance in HAR on two data sets (iSPL and UCI HAR). They acquired 99.06% and 92.13% accuracy on iSPL and UCI HAR data sets, respectively. Mekruksavanich and Jitpattanakul [ 54 ] also employed the CNN–LSTM model for HAR using data from smartwatch sensors from 44 subjects performing 18 activities. They achieved an accuracy of 96.20% using CNN–LSTM, which was better than the performance of CNN and LSTM when the models were used separately. Mekruksavanich and Jitpattanakul [ 55 ] proposed a 4-layered CNN–LSTM model and evaluated its performance using the UCI HAR data set. They found that the CNN–LSTM hybrid model can outperform Vanilla LSTM network, 2-Stacked LSTM network, 3-Stacked LSTM network achieving an accuracy of 99.39% using a 10-fold cross-validation technique. Many other researchers have used CNN–LSTM in HAR to utilize its capabilities of feature extraction and preserving temporal dependencies [ 56 , 57 , 58 , 59 ]. There has been considerable research for HAR which proposed divergent techniques to solve the major challenges, including sensor orientation, sensor placement, and algorithm choice.

In this study, we contributed to this field by evaluating the performance of previously introduced heuristic features [ 26 ] using our data set in intra-position (i.e., senor orientation problem) and inter-position (i.e., sensor placement problem) scenarios using a 1D-CNN–LSTM model. In the original study [ 26 ], the researchers introduced heuristic features to tackle the sensor orientation problem. However, they evaluated the performance of those heuristic features by synthetically introducing orientation in the data set. In our study, we assessed the performance of these features in solving the orientation problem for three different positions, where the sensor orientations were ensured during the data accumulation process. Moreover, we assessed the performance of those heuristic features in solving the sensor placement problem. By doing this, we wanted to inspect if the heuristic features alone can solve the sensor placement problem. In addition, only a few studies adopted the Leave- N -Subject-Out Cross-Validation approach and did it for a considerably small-scale data set. In our study, we adopted the Leave- N -Subject-Out Cross-Validation approach for a data set accumulated from 42 subjects and consisting of over 12 million samples. To be precise, we worked on the following contributions in this study,

We evaluated the effectiveness of previously proposed sensor invariant features’ [ 26 ] performance in the case of sensor orientation problems in HAR for a large-scale data set, where the sensor orientations were practically introduced. Previously, the performance of the heuristic features was evaluated using data, where the orientations were introduced synthetically (intra-position evaluation)

No study in the past evaluated the effectiveness of the heuristic features in solving sensor placement problems. In our study, we assessed the performance of heuristic features in tackling the sensor placement problem in HAR (inter-position evaluation)

We analyzed the performance of the proposed approach in HAR using a Leave-10-Subject-Out Cross-Validation technique for a vast data set containing enormous variations. Previously, most of the studies used Leave-1-Subject-Out Cross-Validation and used comparatively small data sets. Our employed Leave-10-Subject-Out Cross-Validation technique is more challenging for the HAR system than the Leave-1-Subject-Out Cross-Validation technique.

We analyzed the performance of the proposed architecture for six activities with varying intensities (Lying, Sitting, Walking, Running at 3 METs, Running at 5 METs, and Running at 7 METs). We wanted to forge the whole system as practically as possible by introducing sensor orientation, placement problems, and activities with varying intensities.

The rest of the paper is arranged as follows. “ Results ” section describes activity-specific and participant-specific results for both intra-position and inter-position scenarios. “ Discussion ” section discusses our findings, and we conclude our study in “ Conclusion ” section. “ Methods ” section introduces the materials and methods, where we discuss the data accumulation procedures, data pre-processing and feature extraction approach, the architecture of the models, and their workflow. The entire study procedure is depicted in Fig.  2 .

figure 2

Overall workflow diagram of our study

We used data from three positions, and the users had the freedom to keep the smartphone in each position at any orientation. We used the four most common evaluation metrics for multi-class classification studies: Accuracy [ 60 ], Precision [ 60 ], Recall [ 60 ], and F1-Score [ 61 ]. Accuracy is the most suitable metric to present a classification model’s overall performance. The other three metrics are well-suited to describe the model’s performance for the class-specific scenario.

Results for intra-position scenario

In intra-position evaluation, we first analyzed the model’s overall performance for each position. Following, we performed participant-specific and activity-specific analyses.

Overall result

For the intra-position case, the model was trained and tested using the heuristic features corresponding to the same position. We computed results using the Leave-10-Subject-Out Cross-Validation procedure and averaged the test results. The average accuracy, recall, precision and F1-Score for each position are depicted in Fig.  3 .

figure 3

Bar plots with error bars showing averaged evaluation metrics for the intra-position scenario

In the intra-position scenario, we achieved the highest result for the position backpack for every evaluation metric. We recorded 73.64% accuracy, 73.34% recall, 76.83% precision and 72.35% F1-Score for the position backpack. We recorded the second-best results for the position pocket. For the position pocket, the accuracy, recall, precision and F1-Score were 71.46%, 71.07%, 73.66% and 69.82%, respectively. Although the results were lower for the position hand among all the positions, they were not much lower than those for the position pocket. We recorded 70.10% accuracy, 69.98% recall, 72.47% precision and 69.04% F1-Score for the position hand. We hypothesize that we achieved better results for the backpack position, because the smartphone was more stable in the backpack than in the other positions. For activities with high intensities, such as walking or running, the hand frequently moved with the body, which allowed additional variations for the values from the accelerator sensor of the smartphone. Consequently, the values for the heuristic features were affected for the position hand, and the results became lower. If we observe the overall results, the evaluation metrics ranged between 69% and 74% for all the positions. We cannot consider it the best result compared to the previously conducted studies. Still, considering the number of participants, the volume of the data set and the number of sensors, the results seem promising. The accuracies were over 70% for all the positions, which means that the 1D-CNN–LSTM model performed decently as a classification model. The average precision and recall were promising, indicating that our model tried to keep the number of false predictions lower and true predictions higher for each activity class. However, these two metrics will be more meaningful when we observe their value for the activity-specific scenario. The satisfactory F1-Score meant that the 1D-CNN–LSTM model tried to maintain a balanced trade-off between precision and recall.

Participant-specific scenario

We only considered accuracy as a summary metric of model performance for the participant-specific result analysis in the intra-position scenario. We wanted to observe how consistent the model’s performance was for each subject. The accuracy of each participant for each smartphone location is depicted using a line plot in Fig.  4 .

figure 4

Line plot showing accuracies for all participants at every position in the intra-position scenario

For the position pocket, we achieved the highest accuracy of 88.49% for Participant 20. For most participants, the accuracy ranged from 60% to 80%. However, we recorded inferior accuracy in the case of some participants, such as participants 16, 35, 37 and 38. For the position backpack, we recorded the highest accuracy of 90.29% for Participant 27. The accuracy range for most participants was the same as we observed in the pocket case. We also observed inferior performance from the model for some participants, such as participants 4, 16, 22 and 37. For position hand, the highest accuracy was 84.25% for Participant 33. The overall accuracy range was the same as we observed for other positions. Again, the model rendered insufficient accuracy for participants, such as 19, 25 and 37. Considering the overall pictures, for the intra-position scenario, the performance of heuristic features can be regarded as sufficient and propitious. Some participants, including 16 and 37, consistently had low accuracy across all intra-position scenarios. It is somewhat unclear why this is the case, but likely, the result is due to noise in the raw data.

Activity-specific scenario

We also analyzed the result of the intra-position scenario for the activity-specific case. For this analysis, we considered the evaluation metrics such as recall, precision and F1-Score to demonstrate how the heuristic features performed with the help of the 1D-CNN–LSTM model for each activity class. The results for the activity-specific case are depicted in Fig.  5 .

figure 5

Bar plot showing activity-specific results with error bars for the intra-position scenario

First, we will discuss the values of evaluation metrics for the pocket position. We recorded the highest precision of 86.58% for the activity “Walking”. We generally expect a model to generate high precision for all the classes. Our 1D-CNN–LSTM model generated high precision for high-intensity activities such as Walking and Running at 3, 5 and 7 METs for the data of the position pocket. However, the precision for low-intensity activities such as Sitting (61.61%) and Lying (63.70%) was low. This is a well-known result, because sensor signals tend to be small for low-intensity activities; therefore, models misclassify these activities. For the pocket position, all activity classes, except Sitting, had a model-generated recall greater than 70%. We recorded the highest recall of 83.96% for the activity Running at 5 METs. We also expect high recall from a classification model along with high precision. However, our model for the pocket location generated very poor recall (43.76%) for the low-intensity activity of Sitting. The F1-Score in our classification model is a critical metric, because it explains how well-balanced our model is for precision and recall. For the pocket position, the F1-Score was promising for all the activities except for Sitting. For the activity class Sitting, the F1-Score was only 48.65%. The F1-Score was expected to be low for the activity Sitting as we experienced low precision and recall for that same activity. We acquired the highest F1-Score for the activity Walking (82.46%).

For the backpack position, we recorded the highest precision of 87.98% for activity Running at 7 METs. The precision was lower for activities, such as Lying (66.26%) and Running at 5 METs (69.31%). The precision for the activity Sitting was poor for the pocket position; however, for the position backpack, it was improved (76.60%). The recall for the backpack position was similar to the pocket position. We recorded the lowest recall of 55.55% for the activity Sitting. The highest recall was found for the activity of Walking (81.42%). The recall ranged between 70% and 80% for all other activities. The scenario for F1-Score for the position backpack was similar to the position pocket. The lowest F1-Score was recorded for the activity Sitting (61.41%), and the highest F1-Score was recorded for Walking (80.95%). Considering the evaluation metrics for the position backpack, our model seemed to struggle to identify the activity Sitting correctly.

We expected the evaluation metrics for the hand position to be poorer than those for the other positions. This is because, as we mentioned before, the continuous movement of the hand during high-intensity activities causes extensive variations in the data collected from the accelerometer sensor. The precision values for the hand position had a pattern similar to that observed for the pocket position. We recorded the highest and lowest precision for the Walking (87.17%) and Sitting (55.76%) activities, respectively. Regarding recall for the hand position, the scenario was the same as we observed for the pocket position. The recall was highest for the activity Running at 3 METs (81.41%) and lowest for the activity Sitting (50.95%). The recall was poor for the activity Running at 5 METs. The lowest F1-Score for the hand position was recorded for the activity Sitting (51.25%), and the highest F1-Score was for the activity Walking (81.99%). Considering the precision, recall, and F1-Score for all the positions, we found that the model struggled to recognize the activity Sitting for all three positions. For other activities, the model performed well using the heuristic features, especially for the activity Walking.

Results for inter-position scenario

In the case of the inter-position scenario, we performed the same analysis. We will start by discussing the overall results. Following, we will describe the participant-specific and activity-specific results.

Overall results

We trained our model using heuristic features extracted from the raw accelerometer data from one smartphone placement and tested the model’s performance using the heuristic features extracted from the raw accelerometer data of a different smartphone placement. We averaged the evaluation metrics over all the iterations of the validation procedure to calculate the final overall results. The results are shown in Table  2 . The highest accuracy was for the backpack position when the model was trained using the data from the hand position. We recorded 68.66% accuracy, 69.95% precision, 67.07% recall and 64.77% F1-Score in this case. The lowest accuracy result was recorded for the data from the hand position when the model was trained using data from the backpack position. The accuracy and F1-Score were below 60% in this case. When the model was trained using data from the pocket position and tested using data from the backpack position, we acquired results that were almost similar to the case, where the model was trained using the data from hand and tested using the data from the backpack. For other cases, the metrics ranged between 62 and 66%.

We expected to have poorer results in the case of inter-position evaluation, since the training data and test data were from different positions and different participants. The values for the evaluation metrics were below 70%. However, the result seems acceptable considering the simple heuristic features and data from a single accelerometer. The model seemed to perform the best when trained using the data from hand.

Participant-specific result

We only considered accuracy as an evaluation metric for participant-specific evaluation. As mentioned before, the principal purpose of this analysis was to observe the number of participants for whom the model’s performance was poor. The analysis is depicted graphically in Fig.  6 .

figure 6

Participant-specific results for the inter-position scenario

When the model was trained using data from the pocket position and tested using the data from other positions, most participants’ accuracies were above 60%. In addition, the accuracies were consistent for each participant for every position, i.e., for a particular participant, if the model performed well for the data from the hand position, the model performed well for the data from the backpack position. However, there were some exceptions; for instance, for participant 22, the accuracy of the data from the backpack was the lowest (40.71%), but the accuracy of the data from hand was 76.81%. For participant 23, the accuracy was 42.05% when the model was tested using data from the hand position, but for the same participant, the accuracy was 75.97% when tested using data from the backpack position.

When the model was trained using the data from the backpack, for some participants, the model performed very well when the test data was from the pocket. For example, for participants 20, 31 and 33, the accuracies were 86.22%, 83.69% and 88.13%, respectively. However, the scenario was not the same when the test data was from the hand position. The highest accuracy recorded when the test data was from the hand position was 75.91% for Participant 28. When the test data was from the hand position, the accuracy was below 47% for some participants, such as 19, 26 and 35. Similar accuracies were recorded for participants 6, 16, 21, 24 and 37 when the test data was from pocket.

For the final case, where the model was trained using data from hand and tested for two other positions, the results were better for most participants when the test data was from the backpack position. For most of the participants, the accuracies were about 70%. We recorded the highest accuracy of 82.67% for participant 27 when the training data was from the hand position, and test data was from the backpack position. Still, there were some participants, such as 19, 22, 35, and 37, for whom the accuracy was very low. When the test data was from the pocket position, we recorded the best accuracy (78.15%) for Participant 20. For most participants, accuracies were around 60%, except for some participants, such as 21, 35 and 39, where the accuracies were below 50%.

Activity-specific results

For the activity-specific results, we will discuss each evaluation metric for all the cases individually. The evaluation metrics for each inter-position scenario for every activity class are depicted in Fig.  7 .

figure 7

Activity-specific results for the inter-position scenario

For precision, we can see that the lowest values were found for the activity Sitting. For all the inter-position cases, the precision for the activity Sitting was around 50%. We encountered similar results for intra-position cases. The model found it challenging to identify the activity Sitting correctly in both inter-position and intra-position situations. The precision for the activity Walking was satisfactory for all the inter-position cases and was around 80%. For the activity Lying, we experienced precision ranging between 50% and 60%. This means the model mislabelled many samples from other high-intensity activities to Sitting and Lying. For the activity Running at 3 METs, the precision was lower than 60% for two cases, and in both cases, the test data was from the hand position. For other cases, the precision was around 70%. For the activity Running at 5 METs, the precision was approximately 70% when test data was from the backpack position. For other cases, the precision was around 60%. Finally, for the activity Running at 7 METs, the precision was approximately 80% except for two cases. The test data were from the pocket in both cases, and the precision was around 65%. Observing the precision for all activity classes, it is clear that the model had lower accuracy from low-intensity activities, which decreased overall precision.

Regarding the recall for all the inter-position cases, we recorded the poorest performance for two activity classes, Sitting and Running at 5 METs. For the activity Sitting, we recorded poor recall (between 20% and 30%) in three cases. In two of those three cases, the test data was from the backpack position, the other had data from the pocket position as test data, and the training data was from the hand position. In other cases, the recall was between 55 and 70%. Poor precision and recall for the activity Sitting means that the model mislabelled other activities as the activity Sitting and mislabelled many samples from Sitting activities to other activities. Although the precision was around 60% for all the cases of activity Lying, the recall was comparatively better and around 80% for three cases. In two of those three cases, the train data was from hand, and the other case had training data from the pocket position and test data from the backpack position. For Walking and Running at 3 METs, the recall was satisfactory and ranged between 70 and 80% for most inter-position cases. As mentioned before, the recall was poor for the activity Running at 5 METs. The recall (above 60%) was comparatively better for the activity Running at 5 METs only when the test data was from the backpack. For the activity Running at 7 METs, the recall was satisfactory. After observing the precision and recall, one significant finding was that the metrics were always better when the test data was from the backpack position.

In the case of F1-Score, the result was similar to the previous two metrics as it is a harmonic mean of recall and precision. The F1-Score was promising for the activity Walking, at approximately 80%, and Running at 7 METs, at about 70%, for all the cases. The F1-Score was low for the activities Sitting and Running at 5 METs. The low value of F1-Score for the activities Sitting and Running was expected as we experienced low precision and recall for those activities. For the activities Lying and Running at 3 METS, the F1-Score was better for three cases. Among those three cases, two cases had the data from the hand as train data, and in the other case, the test data was from the backpack, and the train data was from the pocket.

Considering all evaluation metrics for all the inter-position cases, we can conclude that the heuristic features with the 1D-CNN–LSTM model struggled with low-intensity activities in both intra-position and inter-position cases. Along with the low-intensity activities, we also found poor performance for the activity Running at 5 METs.

In our study, we wanted to explore the effectiveness of heuristic features in solving sensor orientation and sensor placement problems with the help of a 1D-CNN–LSTM model. We collected data from only a smartphone accelerometer sensor. We had 42 participants, and we followed the Leave-10-Subject-Out Cross-Validation approach. Our study had two types of analysis: intra-position (i.e., sensor orientation problem) and inter-position (i.e., sensor placement problem) analyses. In the intra-position scenario, we checked the effectiveness of the heuristic features on a data set, where the orientations were introduced during data collection. In the study [ 26 ], where the heuristic features were introduced, they evaluated the performance of the heuristic features on five public data sets, which they named A [ 62 ], B [ 63 ], C [ 64 ], D [ 65 ], and E [ 66 ]. These data sets were accumulated using sensors fixed at a constant orientation. Therefore, the data sets had orientation information for only one orientation. Their study claimed that the heuristic features removed the orientational information from the data set. Since the orientation information was removed, the data set simulated the scenario, where the orientation of the sensor did not matter anymore. However, their data set did not represent a practical scenario, where multiple orientations can be present. Since, in our case, different orientations of smartphone accelerometer were ensured, we were able to evaluate the performance of the heuristic features in a practical scenario. In addition, the data set on which [ 26 ] performed the evaluation had a low volume of data. They had the highest number of data windows for data set C (30 subjects); the number of data windows was 10,299, with a 50% overlapping ratio. In our case, we had around 4 million data windows for each position. For 3 positions, the total number of data windows was about 12 million. Moreover, we had 42 participants’ data, which offered more diversity for our data set. Their study followed the Leave-1-Subject-Out Cross-Validation and P-Fold Cross Validation approaches, whereas our Leave-10-Subject-Out Cross-Validation approach ensured a more practical test case, where 10 test participants offered completely unseen data to the model. In their study, three of their five data sets had more than one type of sensor. Data sets B and E only used a single accelerometer as we did. To be brief, our study protocol was more practical and simulated a real-life scenario, ensuring a more reliable evaluation of the heuristic features.

As mentioned earlier, the study [ 26 ] followed two validation approaches: P-Fold Cross Validation and Leave-1-Out Cross-Validation. The Leave-1-Out Cross Validation approach and our Leave-10-Out Cross-Validation approach tested the model using data from participants unseen by the model. They introduced 9 heuristic features and used these features by dividing them into 3 sets. The first set had the first 3 heuristic features, the second set had the first 6 features, and the third one had all 9 heuristic features. They evaluated the effectiveness of these features for 4 different classifiers: Bayesian Decision Making (BDM), K-Nearest Neighbour (KNN), Support Vector Machine (SVM) and Artificial Neural Network. In Table  3 , we have tabulated the best result for each data set using the heuristic features, number of features used, name of the classifier, types of sensors, number of sensor units and the same information in our intra-position cases. We only included intra-position cases, because their study was conducted to solve sensor orientation problem for a fixed position.

We found better accuracies for all the intra-position cases when compared with the accuracies they found for data sets C, D and E. Moreover, from their results, we can see that they found the best result for three of their data sets using only the first 3 heuristic features. Therefore, their study’s findings supported selecting the first four heuristic features based on feature importance. In addition, the satisfactory results we found for intra-position cases depict that the heuristic features effectively solve the orientation problem even for practical scenarios.

In our study, we also presented the performance of the heuristic features in participant-specific and activity-specific scenarios. In participant-specific cases, we found that, for most of the participants at each position, the accuracies were around 70%. There were some participants for which the models’ performance was reduced drastically. Moreover, the participants for whom we found poor performance for the model changed according to the sensor placement. For activity-specific scenarios, in intra-position cases, we found that the heuristic features work better for high-intensity activities. We recorded poor precision, recall, and F1-Score for low-intensity activities, such as Lying and Sitting.

Regarding the inter-position cases, we found comparatively better results when we trained the model using the data from the hand position. The worst result was when we trained the model using the data from the backpack position. One interesting finding is that the model that we trained using the data from the hand position, which encompasses data of comparatively more variations because of the frequent movement of the hand, showed the highest accuracy when using data from backpack or pocket position as test data. On the contrary, the model performed poorly when it was trained using the data from the backpack, which encompasses fewer data variations because of the less frequent movements of the backpack. The difference between the values of the evaluation metrics for intra-position and inter-position cases was approximately 10%. The accuracies for the inter-position cases were around 65%. The model performance was informative, considering we used data from one accelerometer and simple heuristic features. The heuristics features were particularly proposed for solving the orientation problem, and we wanted to find out the heuristic features’ effectiveness in the placement problem. The performance of the heuristic features in inter-position cases indicates that if we fuse the heuristic features with other proposed approaches to solve the sensor-placement problem, then there is a high chance that the performance will increase. In addition, for both intra-position and inter-position cases, if we use other types of sensors, such as a Gyroscope and Magnetometer along with an accelerometer, we may find better results using the heuristic features.

We had some interesting findings regarding the participant-specific and activity-specific scenarios for inter-position cases. We observed that, in some cases, when we had good accuracy for a particular subject’s data from one specific position, we had low accuracy for that same subject’s data from a different position. Such a case was for Participant 22 when the model was trained using the data from the backpack position and tested using the data from the other two positions. Similar to the intra-position scenario, in inter-position scenarios, we observed that the models performed poorly for some participants in every inter-position case, reducing the average accuracy for all cases. In the activity-specific scenario, the findings’ pattern was similar to those for intra-position cases. The heuristic features could not perform well for low-intensity activities, but the result was good for high-intensity activities, especially for Walking. However, among the high-intensity activities, the performance for Running at 5 METs was unsatisfactory.

In summary, the performance of the heuristic features with 1D-CNN–LSTM was promising in both intra-position and inter-position cases. We used data from only one accelerometer and performed a Leave-10-Subject-Out Cross-Validation approach. We tried to replicate a practical scenario for a machine learning model and evaluate the performance of the heuristic feature in such cases. For inter-position cases, using other types of sensors might help. An interesting future study would be to observe how the heuristic features perform if fused with existing or newly collected data designed to solve the sensor placement problem.

Our study had research gaps we would like to explore in future work. We explored the effectiveness of the heuristic features using one type of model. Several other classifiers could be promising, and we will investigate other classifiers’ performance on the same data set in future work. In addition, we have not investigated the effectiveness of time and frequency-domain features on our data set. In the future, we will evaluate the performance of time and frequency domain features and compare the results with those achieved using heuristic features. For our data set, we used signals from a single smartphone accelerometer. We should explore how the results change by including signals from the gyroscope and other smartphone sensors. We can fuse the heuristic features with other proposed techniques for solving the sensor placement problem and evaluate its effectiveness for inter-position cases. We only conducted our study for six activities. In future, we intend to conduct the same study with more activities and variations. However, we think that exploring the effectiveness of the heuristic features for different positions in a practical manner would help other studies have a proper idea about the potency of the heuristic features and develop accuracy using other techniques with these heuristic features.

This study examined whether simple heuristic features could help solve the sensor orientation and placement problems when conducting HAR using a single smartphone accelerometer. Our study used the 1D-CNN–LSTM classifier as it utilizes both CNN’s feature extraction power and LSTM’s information-persisting ability. Our study concludes that the heuristic features adequately solve the sensor orientation problem despite a simple study protocol. We found the best accuracy (73.64%) in solving the sensor orientation problem using the heuristic features when the smartphone was placed in the backpack. When working with sensor placement issues, we acquired the best accuracy (68.66%) when we trained the model using the data from position: hand and tested using the data from position: backpack. In addition, we found the heuristic features to be more effective for high-intensity activities. In future, we want to perform the same study using other machine learning algorithms and present a comparative analysis. Furthermore, we will be fusing other methods that eliminate sensor orientation and placement problems with heuristic features to investigate if the outcome can be improved further or not. We believe that the findings from our study will help other researchers decide how to approach solving sensor orientation and placement problems when using heuristic features. Finally, we hope that the outcome of this study will assist in building a robust HAR model in terms of sensor orientation and position variation in the future.

In this section, we will first discuss the data accumulation process. Following, we will explain the data pre-processing and feature extraction procedure. Then, we will briefly discuss the feature selection approach and describe the 1D-CNN–LSTM architecture we used.

Data accumulation

For our study, we collected data from 42 healthy participants for six different activities with varying intensities: Lying, Sitting, Walking, Running at 3 METs, Running at 5 METs, and Running at 7 METs. We acquired ethical approval from the Memorial University Interdisciplinary Committee on Ethics in Human Research (ICEHR #20180188-EX). Before commencing the data accumulation procedure, each participant had to complete the Physical Activity Readiness Questionnaire (PAR-Q). There were 18 male and 24 female participants. The average age, height and weight were 29 (range = 18–56 years) years, 169.17 cm (range = 143–185 cm) and 68.19 kg (range = 43–95.2 kg), respectively. Each participant performed nine trials to complete the data collection protocol. While performing the trials, the participant carried three Samsung Galaxy S7 smartphones (SM-G930W8) in three locations. The locations were the participant’s right pocket, backpack and right hand. The data accumulation process was 65 min long. The order of the trials with duration is given in Table  4 . Trial 1 is the trial with which the participants started the data collection protocol, and Trial 9 refers to the last trial to be completed.

An android application called Ethica Data [ 67 ] was used to collect sensor data. The application recorded the data from the accelerometer sensor’s X , Y and Z axes embedded into the smartphone. The application continuously recorded the sensor’s value and uploaded the value to the server. During the data collection, the participants were free to keep the smartphone in any arbitrary orientation. We collected the data in an indoor environment. We used a treadmill to accumulate data for walking and running at three speeds. We used the Metabolic Equivalent of Task (MET) to quantify the running intensities or speeds. METs are a ratio of the oxygen consumption rate of a person to the corresponding person’s weight. We preferred MET to walk speed, cadence or stride length to measure the intensity, because those units are prone to generate different expenditures for different persons. We wanted to ensure that the participants performed the activity with the same intensity. The mathematical equation to define the MET is given in ( 1 ):

We selected these particular activities in our study to ensure the presence of the most common daily activities. Besides, few HAR studies combined activity types and activity intensity recognition in their work.

Data pre-processing

We performed data resampling and data imputation on the data set. The optimization technique of the Ethica App did not let the app maintain the same data uploading frequency. As a result, the frequency ranged from 5 to 19 Hz. We upsampled the data set to a constant frequency of 30 Hz to eliminate this data imbalance using a published method [ 68 ]. Another challenge with the data was missing values. Missing data occurred because of the temporal connection loss between the Ethica App and the server. We used linear data imputation to impute missing values. The number of samples for each activity at each position after pre-processing is shown in Table  5 ,

Feature extraction

We extracted 9 orientation-invariant heuristic features using the formula [ 26 ] to address the orientational dependency problem. Since the participants had the freedom to place the smartphones in the pre-determined position in any orientation, we experienced different ranges and patterns in sensor values for different participants, even though they were performing the same activity. Figure  8 shows the differences in patterns and ranges of accelerometer axes due to the sensor orientation, while different participants performed the same activity (Running at 7 METs), keeping the smartphone in their backpacks.

figure 8

Accelerometer axis for different participants performing running at 7 METs (position: backpack)

We extracted the previously introduced 9 orientation invariant heuristic features to eliminate this problem. The formulas to extract the 9 orientation-invariant heuristic features are given below,

\(\overrightarrow {{v_{n} }} = \left( {v_{x} \left[ n \right], v_{y} \left[ n \right], v_{z} \left[ n \right]} \right)\) defines a vector, where \(v_{x} \left[ n \right]\) , \(v_{y} \left[ n \right]\) , \(v_{z} \left[ n \right]\) , were values of the accelerometer x -axis, y -axis, and z -axis, respectively, at any time sample n . \(\Delta \overrightarrow {{v_{n} }} = v_{n + 1} - v_{n}\) and \(\Delta^{2} \overrightarrow {{v_{n} }} = v_{n + 1} - v_{n} ,\) defined first-order and second-order time differences, respectively.

A more detailed explanation of the features can be found in [ 26 ]. Although they introduced the nine features mentioned and used them to eliminate the orientational effect, in a previous study [ 69 ], we found the first four features \(w_{1} ,{ }w_{2} ,{ }w_{3,}\) and \(w_{4,}\) to be the most significant and effective in reducing the orientational effect. Therefore, for our study, we only used the first four features.

Heuristic features for different orientations

From Fig.  9 , we can observe that the 4 heuristic features were able to introduce enough similarity for the feature values, while two different participants placed the smartphone in a backpack and performed the same activity (Running at 7 METs).

figure 9

First 4 heuristic features for different participants performing running at 7 METs (position: backpack)

Visual inspection of Fig.  10 shows that features were able to maintain dissimilarity for the feature values, while two different participants placed the smartphone in a backpack and performed different activities (Sitting and Running at 5 METs). The heuristic features reduced the sensor orientation effect from the sensor values. The four features are also simple to extract, which can reduce computational complexity compared to the other feature-extracting methods with numerous features that need to be extracted to eliminate the orientational problem.

figure 10

First 4 heuristic features for different participants performing sitting and running at 5 METs (position: backpack)

Heuristic features for different placements

We investigated the patterns and ranges of the raw accelerometer values and heuristic features for different smartphone placements. The raw accelerometer values should differ in ranges and patterns for the same activity performed by different participants when keeping the smartphone in different placements. From Fig.  11 , we can observe the dissimilarity in patterns and ranges of raw accelerometer values, while two participants performed Running at 5METs, keeping the smartphone in two different locations (Backpack and Pocket).

figure 11

Differences in range and patterns of accelerometer axes due to different sensor placements

From Fig.  12 , we can observe that the first heuristic feature showed similarities in the values and patterns, while the two participants performed the same activity, Running at 5 METs, by keeping the smartphones in different locations (Backpack and Pocket). For the remaining 3 heuristic features, the similarities in ranges looked promising, but the patterns differed substantially.

figure 12

First 4 heuristic features for different smartphone placements and different participants running at 5 METs

Besides, the heuristic features could maintain the dissimilarity in the feature values and range for different activities (Sitting and Running at 5 METs) performed by different participants, keeping the smartphone in different locations (Backpack and Pocket), as depicted in Fig.  13 .

figure 13

First 4 heuristic features for different smartphone placements and different participants performing different activities

1D-CNN–LSTM architecture

In this section, we will discuss our deep-learning approach. Although the heuristic features tried to reduce the gap between the sensor values for the same activity in different placements (i.e., hand, pocket or backpack), there were still substantial differences among the sensor values for different placements. We used a hybrid 1D-CNN–LSTM architecture to address the sensor placement problem. In our proposed model architecture, there were two significant parts. The first part contained the CNN model, and the second included the LSTM and fully connected layers. The reason behind using CNN was its automatic feature extraction capability. In general, a CNN model takes images or data matrices as input. The convolution layer of CNN applies multiple filters or kernels on the feed images or data matrices and extracts meaningful feature maps. The number of feature maps depends on the number of filters used. If \(n\) number of filters are applied on a single data matrix, then we will get \(n\) number of feature maps, where each feature map will try to extract a distinctive feature for that data matrix. After extracting the feature maps, CNN uses the pooling layer to reduce the size of the feature maps. The average or max pooling layer reduces the feature map’s size. The feature maps can be regarded as the automatically extracted features for the input data matrices. In the convolution layer, we propagate the kernels or filters on the data matrices in two different ways. If we propagate the filters in two directions at a time, we call the model 2D-CNN or conventional CNN. If we propagate the filters in only one direction, we call it 1D-CNN. In general, we use 2D-CNN for images and 1D-CNN for data matrices. As mentioned, CNN can be combined with LSTM to maintain temporal and spatial dependency. In an LSTM model, there can be one or more LSTM layers. Each LSTM layer contains multiple LSTM cells, and each LSTM cell has three gates: Forget gate, Input gate and Output gate. We need to feed data matrices as input to the LSTM model. If a data matrix has n samples, then we can denote the samples as \(t_{i}\) , where \(i = 1,2,3 \ldots n\) . When the Input gate processes any particular sample \(t_{i}\) , the Forget gate decides the information to preserve from the previous sample \(t_{i - 1}\) . The Output gate then combines the information from the Input gate and Forget gate to predict the current sequence or data matrix. If an LSTM model follows a CNN model, then the feature maps generated by the CNN model act as input to the LSTM model. Using the extracted feature maps of the CNN model, the LSTM model can find better temporal dependency for a sequence or data matrix. The output from the LSTM model goes to the fully connected layers made of conventional neurons to make the final prediction. We assumed that the CNN portion of the proposed architecture would be capable of bringing meaningful feature maps, which will help reduce the similarity gap in heuristic feature values observed in the case of non-identical placements of smartphones.

Our proposed 1D-CNN–LSTM architecture was designed as a classification model for classifying human activities. The 1D-CNN–LSTM model contained six convolution layers with 512, 256, 64, 128, 256, and 512 filters, followed by an LSTM layer with 512 LSTM cells. Then, we added four fully connected layers with 100, 28, 64 and 6 neurons. We had average pooling layers after the first, third and final convolution layers with a pool size of 3. We also introduced some dropout layers to reduce the overfitting issue in our model. A more detailed description of the model is depicted in Table  6 . We used an “Adam” optimizer with a learning rate of 0.0001. We determined all the hyperparameters for our deep learning algorithm using the trial and error method.

Validation procedure

There were data from 42 participants. We used 30 participants’ data in the training phase, 10 for testing and 2 for validation in each iteration of our validation procedure. The participant’s data in the validation set were constant, but the training and test data changed as we used Leave- N -Subject-Out Cross-Validation. In our case, the value of N was 10, which made our procedure a Leave-10-Subject-Out Cross-Validation technique. As we had data from 40 participants for the training and testing phase, 4 iterations were required for the whole validation procedure. We had 10 different participants’ data in the test set at each iteration. As mentioned, we decided to inspect two separate scenarios: intra-position (i.e., sensor orientation problem) and inter-position (i.e., sensor placement problem) evaluations. In the intra-position evaluation, the 1D-CNN–LSTM model was trained and tested using the data from the same position. In the inter-position scenario, the model was trained using data from one position but tested using the data from the other two positions. To accomplish the intra-position and inter-position evaluation, we trained the 1D-CNN–LSTM model for a particular position using the heuristic features for the 30 participants in the training set. Then, we computed the evaluation metrics using the data of 10 participants in the test set for all three positions. For instance, if the model was trained using the heuristic features of 30 participants in the training set for the pocket position, then we computed the evaluation metrics using the heuristic features of 10 participants in the test set for all three positions: pocket, hand and backpack. In this manner, both intra-position and inter-position results were accumulated for all three positions. According to our validation approach, we had to train the model 4 times to follow the Leave-10-Out Cross-Validation technique for each position. Since we were conducting our study for 3 different positions, we needed to train the model 12 times. We used an early stopping technique in the training of the 1D-CNN–LSTM model. The early stopping technique was designed, so that the model would stop training if the model’s accuracy for the validation data did not improve within the successive 20 epochs. The 1D-CNN–LSTM model needed the training and test data to be segmented into data matrices or windows, because 1D-CNN–LSTM works with data windows or data matrices. We segmented the training and test data in each iteration using a window length of 65 samples with an overlapping ratio of 98.46%. That means each window had 65 samples, and two consecutive windows had 64 samples in common. We used the window length of 65, because we found it to be both computational and time-efficient in our previous study [ 69 ]. The information for the validation approach is organized in Table  7 .

Availability of data and materials

The data sets generated and/or analyzed during the current study are not publicly available, but we may publish them soon.

Barua A, Masum AKM, Bahadur EH, Alam MR, Chowdhury M, Alam MS. Human activity recognition in prognosis of depression using long short-term memory approach. Int J Adv Sci Technol. 2020;29:4998–5017.

Google Scholar  

Bahadur EH, Masum AKM, Barua A, Uddin MZ. Active sense: early staging of non-insulin dependent diabetes mellitus (NIDDM) hinges upon recognizing daily activity pattern. Electronics. 2021;10(18):2194.

Article   Google Scholar  

Subasi A, Radhwan M, Kurdi R, Khateeb K. IoT based mobile healthcare system for human activity recognition. In: 2018 15th learning and technology conference (L&T). IEEE; 2018. p. 29–34.

Zdravevski E, Lameski P, Trajkovikj V, Pombo N, Garcia N. Importance of personalized health-care models: a case study in activity recognition. Stud Health Technol Inform. 2018;249:185–8.

PubMed   Google Scholar  

Jalal A, Kamal S, Kim D. A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors. 2014;14(7):11735–59.

Article   ADS   PubMed   PubMed Central   Google Scholar  

Zhuang Z, Xue Y. Sport-related human activity detection and recognition using a smartwatch. Sensors. 2019;19(22):5001.

Yin J, Yang Q, Pan JJ. Sensor-based abnormal human-activity detection. IEEE Trans Knowl Data Eng. 2008;20(8):1082–90.

Ha S, Choi S. Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors. In: 2016 international joint conference on neural networks (IJCNN). IEEE; 2016. p. 381–8.

Wang A, Chen G, Yang J, Zhao S, Chang C-Y. A comparative study on human activity recognition using inertial sensors in a smartphone. IEEE Sens J. 2016;16(11):4566–78.

Article   ADS   Google Scholar  

Barna A, Masum AKM, Hossain ME, Bahadur EH, Alam MS. A study on human activity recognition using gyroscope, accelerometer, temperature and humidity data. In: 2019 international conference on electrical, computer and communication engineering (ECCE). IEEE; 2019. p. 1–6.

Attal F, Mohammed S, Dedabrishvili M, Chamroukhi F, Oukhellou L, Amirat Y. Physical human activity recognition using wearable sensors. Sensors. 2015;15(12):31314–38.

Olguın DO, Pentland AS. Human activity recognition: accuracy across common locations for wearable sensors. In: Proceedings of 2006 10th IEEE international symposium on wearable computers, Montreux, Switzerland. Citeseer; 2006. p. 11–14.

Zhang M, Sawchuk A. A feature selection-based framework for human activity recognition using wearable multimodal sensors. In: 6th international ICST conference on body area networks. 2012.

Masum AKM, Barua A, Bahadur EH, Alam MR, Chowdhury MAUZ, Alam MS. Human activity recognition using multiple smartphone sensors. In: 2018 international conference on innovations in science, engineering and technology (ICISET). IEEE; 2018. p. 468–73.

Hassan MM, Uddin MZ, Mohamed A, Almogren A. A robust human activity recognition system using smartphone sensors and deep learning. Futur Gener Comput Syst. 2018;81:307–13.

Ronao CA, Cho S-B. Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl. 2016;59:235–44.

Ronao CA, Cho S-B. Deep convolutional neural networks for human activity recognition with smartphone sensors. In: International conference on neural information processing. Springer; 2015. p. 46–53.

Khan AM, Lee Y-K, Lee S-Y, Kim T-S. Human activity recognition via an accelerometer-enabled-smartphone using kernel discriminant analysis. In: 2010 5th international conference on future information technology. IEEE; 2010. p. 1–6.

Filippoupolitis A, Oliff W, Takand B, Loukas G. Location-enhanced activity recognition in indoor environments using off the shelf smart watch technology and BLE beacons. Sensors. 2017;17(6):1230.

Shoaib M, Incel OD, Scholten H, Havinga P. Smokesense: online activity recognition framework on smartwatches. In: International conference on mobile computing, applications, and services. Springer; 2018. p. 106–24.

Knighten J, McMillan S, Chambers T, Payton J. Recognizing social gestures with a wrist-worn smartband. In: 2015 IEEE international conference on pervasive computing and communication workshops (PerCom workshops). IEEE; 2015. p. 544–9.

Atallah L, Lo B, King R, Yang G-Z. Sensor placement for activity detection using wearable accelerometers. In: 2010 International conference on body sensor networks. IEEE; 2010. p. 24–9.

Davoudi A, Mardini MT, Nelson D, Albinali F, Ranka S, Rashidi P, Manini TM. The effect of sensor placement and number on physical activity recognition and energy expenditure estimation in older adults: validation study. JMIR Mhealth Uhealth. 2021;9(5): e23681.

Article   PubMed   PubMed Central   Google Scholar  

Yang R, Wang B. PACP: a position-independent activity recognition method using smartphone sensors. Information. 2016;7(4):72.

Chen Y, Shen C. Performance analysis of smartphone-sensor behavior for human activity recognition. Ieee Access. 2017;5:3095–110.

Yurtman A, Barshan B. Activity recognition invariant to sensor orientation with wearable motion sensors. Sensors. 2017;17(8):1838.

Rasekh A, Chen C-A, Lu Y. Human activity recognition using smartphone. arXiv preprint. 2014. arXiv:1401.8212.

Alo UR, Nweke HF, Teh YW, Murtaza G. Smartphone motion sensor-based complex human activity identification using deep stacked autoencoder algorithm for enhanced smart healthcare system. Sensors. 2020;20(21):6300.

Siirtola P, Röning J. User-independent human activity recognition using a mobile phone: offline recognition vs. real-time on device recognition. In: Distributed computing and artificial intelligence: 9th international conference. Springer; 2012. p. 617–27.

Zdravevski E, Lameski P, Trajkovik V, Kulakov A, Chorbev I, Goleva R, Pombo N, Garcia N. Improving activity recognition accuracy in ambient-assisted living systems by automated feature engineering. IEEE Access. 2017;5:5262–80.

Guo H, Chen L, Chen G, Lv M. Smartphone-based activity recognition independent of device orientation and placement. Int J Commun Syst. 2016;29(16):2403–15.

Chen Z, Zhu Q, Soh YC, Zhang L. Robust human activity recognition using smartphone sensors via CT-PCA and online SVM. IEEE Trans Ind Inf. 2017;13(6):3070–80.

Ustev YE, Durmaz Incel O, Ersoy C. User, device and orientation independent human activity recognition on mobile phones: challenges and a proposal. In: Proceedings of the 2013 ACM conference on pervasive and ubiquitous computing adjunct publication. 2013. p. 1427–36.

Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In: Ambient assisted living and home care: 4th international workshop, IWAAL 2012, Vitoria-Gasteiz, Spain, December 3–5, 2012 proceedings 4. Springer; 2012. p. 216–23.

Tran DN, Phan DD. Human activities recognition in android smartphone using support vector machine. In: 2016 7th international conference on intelligent systems, modelling and simulation (ISMS). IEEE; 2016. p. 64–8.

Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL. Training computationally efficient smartphone–based human activity recognition models. In: Artificial neural networks and machine learning—ICANN 2013: 23rd international conference on artificial neural networks Sofia, Bulgaria, September 10–13, 2013 proceedings 23. Springer; 2013. p. 426–33.

Bayat A, Pomplun M, Tran DA. A study on human activity recognition using accelerometer data from smartphones. Procedia Comput Sci. 2014;34:450–7.

Uddin MT, Uddiny MA. Human activity recognition from wearable sensors using extremely randomized trees. In: 2015 international conference on electrical engineering and information communication technology (ICEEICT). IEEE; 2015. p. 1–6.

Gupta S, Kumar A. Human activity recognition through smartphone’s tri-axial accelerometer using time domain wave analysis and machine learning. Int J Comput Appl. 2015;975:8887.

Paul P, George T. An effective approach for human activity recognition on smartphone. In: 2015 IEEE international conference on engineering and technology (ICETECH). IEEE; 2015. p. 1–3.

Kaghyan S, Sarukhanyan H. Activity recognition using k-nearest neighbor algorithm on smartphone with tri-axial accelerometer. Int J Inform Models Anal THEA Int Sci Soc Bulg. 2012;1:146–56.

Fan L, Wang Z, Wang H. Human activity recognition model based on decision tree. In: 2013 international conference on advanced cloud and big data. IEEE; 2013. p. 64–8.

Lara OD, Labrador MA. A mobile platform for real-time human activity recognition. In: 2012 IEEE consumer communications and networking conference (CCNC). IEEE; 2012. p. 667–1.

Yang J, Nguyen MN, San PP, Li XL, Krishnaswamy S. Deep convolutional neural networks on multichannel time series for human activity recognition. In: Twenty-fourth international joint conference on artificial intelligence. 2015.

Zeng M, Nguyen LT, Yu B, Mengshoel OJ, Zhu J, Wu P, Zhang J. Convolutional neural networks for human activity recognition using mobile sensors. In: 6th international conference on mobile computing, applications and services. IEEE; 2014. p. 197–205.

Xu Y, Qiu TT. Human activity recognition and embedded application based on convolutional neural network. J Artif Intell Technol. 2021;1(1):51–60.

Wan S, Qi L, Xu X, Tong C, Gu Z. Deep learning models for real-time human activity recognition with smartphones. Mobile Netw Appl. 2020;25:743–55.

Varshney N, Bakariya B, Kushwaha AKS, Khare M. Human activity recognition by combining external features with accelerometer sensor data using deep learning network model. Multimed Tools Appl. 2022;81(24):34633–52.

Almaslukh B, Al Muhtadi J, Artoli AM. A robust convolutional neural network for online smartphone-based human activity recognition. J Intell Fuzzy Syst. 2018;35(2):1609–20.

Inoue M, Inoue S, Nishida T. Deep recurrent neural network for mobile human activity recognition with high throughput. Artif Life Robot. 2018;23:173–85.

Alessandrini M, Biagetti G, Crippa P, Falaschetti L, Turchetti C. Recurrent neural network for human activity recognition in embedded systems using PPG and accelerometer data. Electronics. 2021;10(14):1715.

Murad A, Pyun J-Y. Deep recurrent neural networks for human activity recognition. Sensors. 2017;17(11):2556.

Xia K, Huang J, Wang H. LSTM-CNN architecture for human activity recognition. IEEE Access. 2020;8:56855–66.

Mekruksavanich S, Jitpattanakul A. Smartwatch-based human activity recognition using hybrid LSTM network. In: 2020 IEEE sensors. IEEE; 2020. p. 1–4.

Mekruksavanich S, Jitpattanakul A. LSTM networks using smartphone data for sensor-based human activity recognition in smart homes. Sensors. 2021;21(5):1636.

Muhammad K, Ullah A, Imran AS, Sajjad M, Kiran MS, Sannino G, de Albuquerque VHC. Human action recognition using attention based LSTM network with dilated CNN features. Futur Gener Comput Syst. 2021;125:820–30.

Deep S, Zheng X. Hybrid model featuring CNN and LSTM architecture for human activity recognition on smartphone sensor data. In: 2019 20th international conference on parallel and distributed computing, applications and technologies (PDCAT). IEEE; 2019. p. 259–64.

Khatun MA, Yousuf MA, Ahmed S, Uddin MZ, Alyami SA, Al-Ashhab S, Akhdar HF, Khan A, Azad A, Moni MA. Deep CNN-LSTM with self-attention model for human activity recognition using wearable sensor. IEEE J Transl Eng Health Med. 2022;10:1–16.

Challa SK, Kumar A, Semwal VB. A multibranch CNN-BiLSTM model for human activity recognition using wearable sensor data. Vis Comput. 2021;38:1–15.

Tharwat A. Classification assessment methods. Appl Comput Inform. 2020;17(1):168–92.

Sasaki Y. The truth of the F-measure. Teach Tutor Mater. 2007;1(5):1–5.

Barshan B, Yüksek MC. Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. Comput J. 2014;57(11):1649–67.

Ugulino W, Cardador D, Vega K, Velloso E, Milidiú R, Fuks H. Wearable computing: accelerometers’ data classification of body postures and movements. In: Brazilian symposium on artificial intelligence. Springer; 2012. p. 52–61.

Anguita D, Ghio A, Oneto L, Parra Perez X, Reyes Ortiz JL. A public domain dataset for human activity recognition using smartphones. In: Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning. 2013. p. 437–42.

Zhang M, Sawchuk AA. USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In: Proceedings of the 2012 ACM conference on ubiquitous computing. 2012. p. 1036–43.

Casale P, Pujol O, Radeva P. Human activity recognition from accelerometer data using a wearable device. In: Iberian conference on pattern recognition and image analysis. Springer; 2011. p. 289–96.

Data E. Ethica [Mobile app]. 2020.

Smith J, Gossett P. A flexible sampling-rate conversion method. In: ICASSP'84 IEEE international conference on acoustics, speech, and signal processing. IEEE; 1984. p. 112–5.

Barua A, Fuller D, Musa S, Jiang X. Exploring orientation invariant heuristic features with variant window length of 1D-CNN-LSTM in human activity recognition. Biosensors. 2022;12(7):549.

Download references

Acknowledgements

Not applicable.

This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), Grant Number RGPIN-2020-05525.

Author information

Authors and affiliations.

Department of Computer Science, Faculty of Science, Memorial University of Newfoundland, St. John′s, A1B 1V6, Canada

Arnab Barua & Xianta Jiang

Department of Community Health and Epidemiology, University of Saskatchewan, Health Science Building, 107 Wiggins Road, Saskatoon, SK, S7N 5E5, Canada

Daniel Fuller

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: AB, XJ and DF. Data curation: DF. Formal analysis: AB. Funding acquisition: XJ and DF. Investigation: AB. Methodology: AB. Supervision: XJ and DF. Writing: AB. Review and editing: AB, XJ and DF.

Corresponding author

Correspondence to Xianta Jiang .

Ethics declarations

Ethics approval and consent to participate.

The Memorial University Interdisciplinary Committee on Ethics in Human Research (ICEHR #20180188-EX) granted us the ethical permissions to collect and use the necessary data set. The research was carried out in accordance with the Declaration of Helsinki.

Consent for publication

We acquired the necessary consent for publication from appropriate individuals.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Barua, A., Jiang, X. & Fuller, D. The effectiveness of simple heuristic features in sensor orientation and placement problems in human activity recognition using a single smartphone accelerometer. BioMed Eng OnLine 23 , 21 (2024). https://doi.org/10.1186/s12938-024-01213-3

Download citation

Received : 22 August 2023

Accepted : 22 January 2024

Published : 17 February 2024

DOI : https://doi.org/10.1186/s12938-024-01213-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • 1D-CNN–LSTM
  • Metabolic Equivalent of Tasks
  • Accelerometer sensor
  • Human activity recognition
  • Sensor orientation
  • Sensor placement

BioMedical Engineering OnLine

ISSN: 1475-925X

problem solving for evaluation

IMAGES

  1. How to Write a Problem Solution Essay

    problem solving for evaluation

  2. 7 steps to master problem solving methodology

    problem solving for evaluation

  3. 5 Problem Solving Strategies to Become a Better Problem Solver

    problem solving for evaluation

  4. What are the problem solving steps?

    problem solving for evaluation

  5. Problem Solving Framework Organisational Assessment

    problem solving for evaluation

  6. Collaborative Problem-Solving Steps

    problem solving for evaluation

VIDEO

  1. Problem Solving

  2. PROBLEM SOLVERS

  3. Problem solving

  4. problem solving

  5. problem Solving Skills

  6. Measuring Large Language Models' Cognitive Behavioral Reasoning for Cybersecurity

COMMENTS

  1. 31 examples of problem solving performance review phrases

    April 1, 2022 - 7 min read Share this article Jump to section You're doing great You should think of improving Tips to improve Use these practical examples of phrases, sample comments, and templates for your performance review, 360-degree feedback survey, or manager appraisal.

  2. Problem Solving: 40 Useful Performance Feedback Phrases

    Problem Solving: Self Evaluation Questions How well do you solve issues and are you confident in your abilities? Give an instance you solved a problem, and it was successful.

  3. Self Evaluation Comments for Problem Solving (30 Examples)

    Self-evaluation is an essential aspect of professional development. It helps you to identify areas of improvement and measure your progress towards achieving your goals. By evaluating your problem-solving skills, you can identify your strengths and weaknesses and take steps to improve your performance.

  4. Problem Solving Skills: Performance Review Examples (Rating 1

    Problem solving is an important skill in any work environment: it includes the ability to identify, understand, and develop solutions to complex issues while maintaining a focus on the end goal. Evaluating this skill in employees during performance reviews can be highly beneficial for both the employee and the organization.

  5. What is Problem Solving? Steps, Process & Techniques

    1. Define the problem Diagnose the situation so that your focus is on the problem, not just its symptoms. Helpful problem-solving techniques include using flowcharts to identify the expected steps of a process and cause-and-effect diagrams to define and analyze root causes. The sections below help explain key problem-solving steps.

  6. 174 Performance Feedback Examples (Reliability, Integrity, Problem Solving)

    Problem Solving and Decision Making The performance review assesses an employee's ability to analyze complex problems, evaluate options, and make sound decisions within a reasonable timeframe. Looks at how well they troubleshoot issues.

  7. PDF THIRTEEN PROBLEM-SOLVING MODELS

    1. Clarify Explore the Vision Identify your goal, desire or challenge. This is a crucial first step because it's easy to assume, incorrectly, that you know what the problem is. However, you may have missed something or have failed to understand the issue fully and defining your objective can provide clarity.

  8. 35 problem-solving techniques and methods for solving complex problems

    Problem-solving methods are primarily designed to help a group or team through a process of first identifying problems and challenges, ideating possible solutions, and then evaluating the most suitable.

  9. The Problem-Solving Process

    1. Identifying the Problem While it may seem like an obvious step, identifying the problem is not always as simple as it sounds. In some cases, people might mistakenly identify the wrong source of a problem, which will make attempts to solve it inefficient or even useless.

  10. (PDF) Evaluation process and problemsolving

    Evaluation process and problemsolving DOI: CC BY 4.0 Authors: Gaetano Martino Università degli Studi di Perugia Abstract and Figures The paper aims at investigating some aspects of the...

  11. Evaluating Problems

    Evaluating Problems This course is part of Solving Complex Problems Specialization Taught in English 20 languages available Some content may not be translated Instructor: A/Professor Iain Hay Enroll for Free Starts Feb 17 Financial aid available 10,472 already enrolled Included with • Learn more About Outcomes Modules Recommendations Testimonials

  12. 55 Effective Performance Review Phrases to Use

    Communication. Teamwork and Collaboration. Goal Achievement. Creativity, Problem-Solving, and Critical Thinking. Flexibility and Adaptability. Time Management and Dependability. Leadership and Management. 3 Things to Avoid When Writing Performance Reviews. Performance Reviews Are Just One Piece of Performance Management.

  13. 45 Examples of Effective Performance Evaluation Phrases

    Leadership. "You are fair and treat everyone in the office as an equal.". "You lead by example. Your approach of embracing change and adapting to changing work situations encourages others to do the same." "Your team consistently meets their goals often exceeding expectations.".

  14. 240 Useful Performance Review Phrases for Better Employee Evaluations

    Example Phrases about Creative Problem Solving Strengths. Displays the ability to come up with innovative solutions to complex problems. Applies creative thinking with creative solutions to specific tasks. Possesses interpersonal skills for solving problems with others. Is able to come up with multiple solutions when the opportunity arises.

  15. 100 Useful Performance Review Comments To Choose From!

    So, let's dive into some performance review comments related to problem-solving skills. Positive: "Demonstrates excellent problem-solving skills, able to analyze complex problems and develop effective solutions." "Approaches problems systematically and logically, using a variety of resources and tools to find solutions."

  16. Evaluation model of problem solving

    Problem solving is a style of thinking, which transforms a given problem to the goal state through a so-called PS (problem solving) path.Different from the traditional GPS (General Problem Solver) approach, the focus in this paper is placed on how to judge the performance of PS paths, that is, the evaluation problem of problem solving.

  17. Problem Solving Performance Review Phrases Examples

    Tenacious and stubborn, there are few problems he can't solve. John has come over to say hi and ended up solving his problem. More often than not he was the go-to man for problem-solving. John expertly came through and solved each and every problem. John has solved his problem more than once when no one could.

  18. 100 Useful Phrases for Employee Performance Reviews

    Use these performance review phrases to rank the problem-solving talents and abilities of your employees: 70. X demonstrates a methodical approach to problem-solving. 71. X is never short of creative solutions. 72. X converts severe problems into compelling opportunities. 73.

  19. 300 Performance Review Phrases (2023 Edition)

    Problem-Solving Skills Strengths 258. Sees problems as challenges to be overcome. 259. Brings ideas to the table when discussing problem-solving 260. Is able to pressure test his/her own ideas in a calm but assertive way 261. Is able to persuade others that [his/her] solutions are the best course of action 262.

  20. The Performance Review Problem

    The Performance Review Problem. As the arcane annual assessment earns a failing grade, employers struggle to create a better system to measure and motivate their workers. After an annual review ...

  21. Adaptability: 40 Useful Performance Feedback Phrases

    Adaptability: Meets Expectations Phrases. Incorporates feedback effectively and deals positively with praises and criticism. Understands, negotiates and balances diverse views and beliefs, particularly in multicultural environments, to reach workable solutions. Pushes aside the work planned for the day to respond to an emerging problem.

  22. 142 Employee Performance Evaluation Examples (Initiative, Flexibility

    Employee performance evaluation is a process used to assess an employee's job performance and to make recommendations for improvement. Evaluation results should be used to help the employee and/or supervisor improve their working relationships.

  23. Initiative: 40 Useful Performance Feedback Phrases

    Initiative: 40 Useful Performance Feedback Phrases Initiative: Use these sample phrases to craft meaningful performance evaluations, drive change and motivate your workforce. An initiative is the ability to assess and initiate things independently often done without any managerial influence offered. Initiative: Exceeds Expectations Phrases

  24. Introducing Gemini 1.5, Google's next-generation AI model

    Relevant problem-solving with longer blocks of code. ... When tested on a comprehensive panel of text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing our large language models (LLMs). And when compared to 1.0 Ultra on the same benchmarks, it performs at a broadly similar level. ...

  25. The role of provincial health administration in supporting district

    The methodological approach. Our overall methodological approach is realist evaluation (RE). RE is a theory-driven evaluation approach that seeks to understand why and how a programme works, for whom, and in what circumstances [19, 20].By answering these questions, RE attempts to provide a plausible causal account of how the interaction of actors with an intervention triggers mechanisms that ...

  26. [2402.10104] GeoEval: Benchmark for Evaluating LLMs and Multi-Modal

    Recent advancements in Large Language Models (LLMs) and Multi-Modal Models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a ...

  27. Self Evaluation Comments for Problem Solving (30 Examples)

    Self-evaluation is an essential aspect of professional development. It helps you to id areas of improvement and measure your progress towards get your goals. By evaluating your problem-solving skills, you canister identify your strengths and weaknesses plus carry steps to enhanced your performance. Problem Solving Self-Evaluation Talk Examples I was able to identify who root...

  28. GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on

    This benchmark facilitates a deeper investigation into the performance of LLMs and MMs on solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67\% accuracy rate on the main subset but only a 6.00\% accuracy on the challenging subset.

  29. The effectiveness of simple heuristic features in sensor orientation

    For intra-position evaluation, we acquired 70-73% accuracy; for inter-position cases, the accuracies ranged between 59 and 69%. Moreover, we performed participant-specific and activity-specific analyses. We found that the simple heuristic features are considerably effective in solving orientation problems.