What were the measurable outcomes from the project that demonstrate progress or success?
Determine the models accuracy - overall the model was around 75-80% accurate when comparing the models predictions to what the repair order quality team actually did.
Accuracy by cost - the model was able to determine the correct result with a high level on confidence on RO's with the a cost less than $500. Our biggest risk when replacing the repair order quality team is invoicing the customer the incorrect amount which results in either over charging the customer or under billing them and missing out on revenue. The 923 was able to demonstrate the accuracy in costs buckets and the possible dollar risk with each bucket. While we did not move forward with replacing the quality team this has set the framework to review orders at certain cost thresholds to reduce risk but still have a significant impact.
Model score - the model will generate a score for 100% of the RO's that are passed through. With this score we are able to determine if we should display the message to the end user or leave it alone. If the model is not confident in the results then we can choose not to display the result.
Failure reasons - Providing not only the score of the RO and if it was a failure but stating where the error occured and why it was generating. The 923 delivers this on every RO that is passed through to them. Comparing with notes left by the quality team we have achieved an almost exact match and caught errors they have missed. By producing these failure reasons this is a significant feature when generating the quality check message to technicians. Rather than simply stating something is wrong, double check your work (which they would likely ignore) we are able to generate a check that say X thing is wrong with this section of the order.
Quality check API - By developing this API we now have a mechanism to get results from the model and share them with the end user. In our case this will be when the technician completes a job we will pass the order details to the API to return the score and results. Implementing this mechanism will have a siginificant impact on the overall quality of the RO's because we are implementing a check at the source. In our current state when a technician completes an RO it can take the quality team 6-8+ hours to review the order. This check is now happening in seconds while the work is still fresh in the technicians mind. If the quality team finds an error they will flag it and send back to the technician who will correct, which can take an additional 4-6 hours. With this check happening at the time of completion we should see an overall decrease in time spent reviewing orders by the quality team, time to detect errors, and reduce time needed to complete and invoice the customer.
Describe their project management. Did they deliver items on time? How did they respond to your needs?
The 923 team always met timelines for deliverables and quickly responded to my teams needs. Because they took the time to get to know our business and learn our process they were not only able to adjust to changes in scope but were also able to recommend ways to implement solutions.
They set up our weekly touchpoints and were always available to schedule additional time as needed. They shared constant updates on the progress and results of their work. One of my favorite things about them was their ability to share the results of their work in a clear way that could be understood even if you are unfamiliar with AI/ML. This was my teams first time working with AI/ML and they were able to eliminate any confusion we had on the process/results which can be very complex.
The team always escalated potential blockers early on the process to give my team time to action. If we were delayed in actioning these blockers they continued to work and provided creative short term solutions while our plan was implemented. I greatly appreciate their commitment to the project and the way they were able to adjust on the fly when needed to deliver the best product. They are committed to providing the best experience to their customers and have always delivered on time.
What was your primary form of communication with NineTwoThree?
- Virtual Meeting
- Email or Messaging App
What did you find most impressive or unique about this company?
Their ability to learn about our business and understand our goal was truly impressive. Going from knowing nothing about a company and the way they work to delivering the results they did was impressive.
As I have mentioned this was our first time working with AI/ML so we really didnt know what was needed or what to expect. This was epsecially true in how our data was structured. What I found to be unique about 923 was their willingness to work with us while we built our historical data. It started with manual data queries and uploading to a shared filed before automating this process. Even during the early days of the project they were able to deliver fantastic results and learn more about our business. Because of the process we implemented with 923 we have set ourselves for future work in the AI space.
The final thing I was impressed with is their communication and ability to summarize results. We were constantly updated on their progress. They were able to provide great detail on how they acheived the results without causing confusion and answered any questions we had clearly. The AI/ML world an be overwhelming and complex but when talking through it with them I felt I had all the information necessary to make a decision and provide direction.
Are there any areas for improvement or something NineTwoThree could have done differently?
No the only problem we faced during our project was data. Mainly that we had not set up our data to track historical changes over time making it difficult to analyze. We have learned so much from this team and our currently working on projects to improve our data for future work with AI. Would love to work with this team again.
