Expert profiles helped differentiate partners, but comparison was slow and lacked trust signals; leaving merchants uncertain, limiting expert credibility, and weakening platform governance. I designed a multi-dimensional reputation system that captures merchant feedback at the right moments in the job lifecycle, accelerating first contact and job creation, increasing merchant trust, enabling expert ranking, and clearly signaling partner expertise through service-specific ratings.
Problem
While profiles were a good start to differentiate Experts (Shopify freelancers), comparing profiles was
time-consuming and lacked trust signals for the merchant to make a confident decision. In turn, experts couldn’t
create clout and Shopify lacked the ability to govern the platform efficiently.
Solution
A multi-dimensional reputation system that captures merchant experiences at the right moment of their journey,
and respects the core loops of successful and failed jobs.
Impact
Acceleration of first contact and job creation
Increased trust and expectations by merchants
Creation of a ranking system for better expert surfacing
Clear signals of the expertise of partners via service specific ratings
Multidimensional reviews
Jobs can be complex, so we provided a low cognitive glance of its aspects. Experience through written text, cost
expectations, what other services the job might have entailed, and a judgement of the dimensions of a service:
communication of the expert throughout the job and the quality of their final work.
The final solution featured a two-dimensional score visible via tooltips
Ranking system
The reputation system was key to the creation of ranking in the platform. We used the different
triggers that are hit through the system’s flow to obtain explicit data like job completion and user ratings, as
well as implicit data like fizzled jobs and churned communications.
The star rating was not the only data point affecting ranking
Service expertise
The reputation was service-aware, allowing merchants to clearly observe where an expert had most
of their experience (jobs completed) and where their expertise is (ratings).
Reputation was service-specific, reflecting each expert's skills and services
What is a job?
The main challenges of the reputation system was to determine what a an experience was, in other words, what a job
was. For that, we needed to answer these questions:
When does a job start? When does a job end?
What is the green path? What are other paths a job can take?
When is the right moment to ask about the merchant’s experience?
The back-end of communication system was complex, so we needed to illustrate clearly the merchant’s flow
through a job to confidently answer those questions and decide where each reputation trigger would reside.
Iteration
Our first iteration was purely theoretical which even passed validation testing. Only after release we
were able to observe real behaviours, which shone a light on issues in the system. From there we were able to
pivot quickly and make empirical changes.
Expectation vs reality of the flow of a job
Quickly pivoting
Unforseen issues lead to system gaming and feelings of unfairness. We had to let go of the
expected job journey, but instead ask ourselves “What makes a merchant entitled to a review?”, thus changing
what we consider a job started and a job ended.
what the job experience looks like in detail in our system
Explicit triggers
The reputation system relied on explicit actions and input by the merchant. So we needed these inputs to:
Appear at the right time - when a full experience has concluded and the merchant is not interrupted.
Appear in the right place - Noticeable and understandable within the context
Strike a balance - require enough effort to obtain valueable information, but not too much that becomes a chore
for the user.
Striking a balance for reviews
Jobs are complex and unique to each merchant. We wanted to obtain as much relevant information as possible to
bring value back to the top of the funnel: other merchants looking for an expert to work with.
Here you can see that our initial explorations required a lot of input. The intention was to bring objectivity to
reviews by capturing information first from experts then the merchants. Instead, it created a bottleneck on the
expert’s side.
As we iterated, we were able to pair down the amount of information we were asking for. We had removed the
bottlenecks and reduced the overwhelming feeling of a large form by displaying everything without pagination.
We started exploring different interactions, like dismissing the review and attuning subjectivity by procedurally
revealing content. But from experience with the App Store reviews, we knew that merchants are easily overwhelmed.
Thus, we needed to continue pairing down the content to reduce the visual noise and perceived complexity.
The final design succeeded in asking for the right amount of information without overwhelming the user. It
prioritized required inputs to be able to obtain value when done at its minimum. It proved to be timely by
appearing at the end of the job experience, and in the area where the focus of the merchant was (main communication
area of a job’s page)
