ProdPerfect

A "Software Testing Team as a Service"

Comments: I had a very positive, overall experience with ProdPerfect's product and team members. My aim was to achieve frontend test coverage without hiring a full time QA engineer and I was able to do this with ProdPefect and the cost was about 50% of what a QA engineer would have cost. Considering the comparison to a fulltime engineer, the rest of the Continuous Integration system comes for "free" in the sense of having the hourly testing compute engaged + a pass fail dashboard for viewing results.

Pros:

ProdPerfect was exactly what we needed as a small software business that did not want to hire a full time front-end QA engineer. Onboarding was easy because their software automatically measured the user flow in our web app, which is then used to create test cases. The PP team created the test cases and get them up and running without very much input from me. Once testing was operational, we held a highly effective weekly meeting series to discuss the system use, testing issues, and new test cases.

Cons:

The ProdPerfect team has good handle on what is needed in the roadmap. In general it would be ideal for the test execution cases to be run in parallel with no-dependency executions. This is something they have been working on but is not ready yet. Not having this feature means that testing of each case is done in series which takes more time and there are some reporting challenges with failures. For example the JavaScript environment is not aware of which test case is running, so the test case ID cannot be included in logging reports (e.g. Sentry)

Unbelievable customer success team meets an incredibly valuable product offering

Comments: The experience has been extremely positive. Every step of the sales journey and onboarding has been surprisingly delightful: we were offered a free trial for our first month _after_ the tests had been written and integrated into our CI/CD tool, the account manager on our team has been down to Earth and accommodating to some of our shifting needs, and their tech team has been really responsive to feedback on how the integration could be improved and worked with us on testing out new features which we liked. It's also worth noting that ProdPerfect hangs out in a shared Slack channel with us, which has a bot reporting test runs that pass (or sometimes fail, whoopsies). In that channel we can click on a report to view a video of what went wrong, talk to the ProdPerfect team to communicate if it was an intentional change or not, and pretty quick any changes to our tests can get rolled out and we'll kick off another run. It really feels like we have an integrated QA team that has successfully caught a handful of issues for us before they make it to production. Definitely worth it in my opinion.

Pros:

The worst part about setting up an outsourced QA team is training them on the flows users go through - simultaneously how users behave and where they are interacting with the product. The highest value proposition of ProdPerfect to me is the tracking beacon that lets them not only figure out what flows to monitor from the start, but as we launch feature changes, ProdPerfect notices the change in behavior and builds out the next suite of tests. It's incredibly hands off but offering us continued and accurate protection.

Cons:

The data viz tool to look at how users are behaving on the site could use some tuning. It's a bit information overload and unintuitive to manipulate to make it useful. Luckily, I have never had to actually use it and after talking to the team at ProdPerfect I was relieved to know I didn't ever actually have to use it - and they're working on a better way to represent the heaps of data they have on our user behavior already.

Remove regression from your list problems

Comments: ProdPerfect saves us an FTE in QA. They're responsive and well-integrated with the team. We ship bugs occasionally, but not in the code that ProdPerfect covers which, by design, is the highest usage parts of the product.

Pros:

PP removes a class of problem from the list of things I need to "do something" about. Regression is a no-win situation for eng management. Do it well and nobody notices. Screw it up and you never live it down.

Cons:

Lack of UI/visibility. If you deal with regression internally you have the same problem ... that you're doing a lot of work that doesn't visibly contribute to forward movement. Buy ProdPerfect and the only visible indicator of effort is a product that works.

ProdPerfect is our test automation team

Comments: We have been very satisfied, and our partnership with ProdPerfect has allowed us to move quickly and maintain a high level of stability and quality in our user-facing web applications.

Pros:

ProdPerfect has offered us an excellent hybrid of tools + services. Their data collection 'secret sauce' made it possible for them to quickly develop a front-end test automation suite that covers the majority of actions that our users take with our web-facing dashboards, and to update these tests as our products evolve. Their delivery team has been attentive and proactive, iterating and tightening up the test logic to handle the inevitable flaky tests that come with front-end test automation. The overall effect is that we get wide test coverage, that we can run with every deployment to staging & production, and that enables us to deploy frequently and make bold changes to the underlying software, with ProdPerfect as our end-to-end safety net.

Cons:

As with any test automation tool, there are false alarms. But the ProdPerfect team has responded quickly and independently to these, mitigating this concern. It has taken longer than we'd like to bring up new test suites. Part of this is the process of data collection, to build up a map of what real users are doing on our sites. On the positive side of this, they have required surprisingly little direction or guidance in building up the tests, which has allowed us to keep our focus on building new features.

The no-brainer, cost-effective solution for our QA

Comments: The pros section probably summarizes it best – but they are high quality tests that are maintained entirely by the ProdPerfect platform that cover our highest trafficked workflows and are kept up to to date. They are integrated perfectly into our pipelines and provide a final layer of safety in the deployment process at a cost that is significantly less than we would have spent to do it ourselves.

Pros:

- Setup is hands on where you want it to be, hands off where you want it to be. ProdPerfect determined which tests would be the most effective, wrote and delivered them to us, but we got to decide exactly how they fit into our pipelines. - Test maintenance and writing is completely automatic – we know the outcome of the tests that are written but don't need to be involved in the creation or upkeep of them. - Tests are high quality – important workflows have been saved on multiple occasions now from being broken due to the suite catching issues that weren't caught by our unit tests. - Communication is top notch – very professional, quick to respond, friendly. - Multiple times cheaper than doing it ourselves. We would have either had to hire QA or have existing engineers context switch to writing regression tests alongside regular development and unit tests.

Cons:

This sounds ridiculous but we really don't have any. We got exactly what we expected.

Data driven testing made easy

Comments: Building out and maintaining an automated E2E front end test system is a full time job for at least one person, ideally more. We needed to keep our engineering costs down as much as possible, so ProdPerfect is a good fit. Their implementation process includes a mapping exercise that identifies ideal test environments and integration points in our CI/CD pipeline. After installing the tracking snippet into production, They were able to identify key E2E user flows and build out tests independently. They hit a few unexpected blockers, but the recorded videos made them easy to troubleshoot.

Pros:

The continuous measuring of our product and the automatic creation and maintenance of test cases as we build out new features is what we like the most about ProdPerfect.

Cons:

The mission control dashboard has expanded and improved quite a bit since I first used ProdPerfect. I would like high level user metrics & observed average page load times from production and test runs to be captured and shown on the dashboard.

Achieving Continuous Testing Effortlessly

Pros:

ProdPerfect has helped our engineering team to move even faster. Our regression test coverage is now based on real usage patterns and regularly being updated and monitored and very well integrated with our CI/CD pipeline. We are able to run the entire test suite in any environment pretty easily and in a timely manner. If and when the tests fail, it is easy to debug and find the cause. The customer service and support from ProdPerfect is top notch.

Cons:

In some rare instances there are false alarms but I'd rather have occasional false alarms than letting a defect or a bug go to production.

ProdPerfect in a CI/CD Pipeline

Comments: Rock solid. The implementation team is great, quick turnarounds, open to working together. It's a partnership - not a product.

Pros:

ProdPerfect is my silent production guardian. We have it installed in our CI/CD pipeline and we run their auto-generated test suite on every commit. When that test suite fails (which means that we have bad code) - it stops our build process and alerts us that we've done something stupid - thus saving us from releasing broken code to production.

Cons:

It's a struggle to come up with something that I don't like - instead, I would offer a small suggestion - in the onboarding process, their test suite needs to record how your users use your app so it can generate tests. It would be nice if there was a way their test suite could crawl your app or you could give it something like Gherkin test writeups and it could generate off that. That would get the time to value faster.

Prodperfect is the perfect solution

Comments: Extra safety net for user experience when doing deep technical work and large refactors, this net is appreciated in times of growth with lots of new engineers who might not realize when they are tripping over dependencies within the sytem. The PP team is responsive and amazing to work with.

Pros:

They help reduce risk of outages without adding bottlenecks to CI/CD as most similar solutions do.

Cons:

Prodperfect has a lot of opportunity to lead in the space with new features. Would like to see more sooner!

Managed automated end-to-end testing that really works

Comments: The primary problem we are solving is code quality and speed of high-quality software development. By having good end-to-end tests, we identify potential problems with new code before it is merged/can be released, and we catch it at the point that the developer who wrote it can quickly identify and fix it. (As opposed to waiting until it's merged, and then having a developer who may not have written it try to find it/figure it out).

Pros:

Our developers no longer need to write and maintain the end-to-end tests that made them less efficient. Additionally, the e2e tests from ProdPerfect are better and cover more cases, and they're updated more quickly. Much better than doing it ourselves.

Cons:

Test updating on weekends/after hours does not happen, so you sometimes get stuck not being able to merge on the weekends if you have changes to interface.

Focused Automation on What Customers Use Most

Comments: We had a need to run continuous regression automation in the background to ensure site reliability (at a user level). The ProdPerfect team was able to come in, analyze user behavior (at the code level) and create a custom test suite that fit our needs. I now get hourly results and alerts when something appears to be impacting the user experience.

Pros:

The test suite developed for our products is based on the features/functionality that our customers actually use. Rather than "automating everything" and wading through the results. We can focus on alerts generated that impact the majority of users. Also, the ProdPerfect team is very responsive to questions and concerns that come up. Response times can be measured in minutes, rather than hours.

Cons:

Running the test suite on our local machines is time consuming and there is not a mobile testing option. Ideally we would have a "dashboard" to allow us to easily select the tests we want to run and kick them off with the click of a mouse.

Exactly what we were looking for

Pros:

With nearly no effort, we started end-2-end frontend testing for our application. Test-results are very helpful and we were able to discover some regression without any manual effort from our team.

Cons:

It requires installing a tracking snippet - this was a bit challenging to align with our data-privacy regulations, but finally we managed to do ...

Review of ProdPerfect by n2y

Comments: I have an enjoyable experience with ProdPerfect with great rapport with the team

Pros:

I enjoyed using mission control. I appreciate the communication from the ProdPerfect team

Cons:

There were not parts I disliked. If I ever had any questions, they answered them thoroughly

Prod Perfect is like having a hidden co-worker

Comments: We are a smaller company with one QA versus 7 developers. Naturally the time we have to test is limited. It's nice to know the tests are running hourly and two different environments, and time a failure has happened, we get a quick update from their contact person. It's like having a co-worker instead of a vendor!

Pros:

I like the easy of on-boarding this product to our online applications. It was easy for us to have them switch running their tests between our test environment and production environment. Each time I've had a question or concern, they are super responsive and complete in their answers and explanations.

Cons:

There's little to dislike. The test replay is easy to access. Having the test scripts readily available is something I'm still working on.

Extra piece of mind

Comments: The team at ProdPerfect really care. From the first calls until full implementation, they are interested in learning all they can about your product and are eager to help in any way. ProdPerfect adds an additional layer of testing to your suite to help ensure you are always shipping working code.

Pros:

With ProdPerfect we can plug our regression testing right into our CI tool so we can prevent shipping our code that has bugs. In addition to automated testing you’ll also have a real engineer actively monitoring your tests to ensure they are accurate and up to date.

Cons:

Nothing so far! Everything has been great.

Trusted team, working hand in hand

Pros:

Super professional team, great support, and dedication.

Cons:

Nothing really. I can't think of any shortcomings in their service.

ProdPerfect has been great for our office.

Comments: ProdPerfect gas been quick to respond to any need or inquiry. ProdPerfect testing has made a big difference for us.

Pros:

Running the automated tests is easy and bug reports generated are precise and detailed.

Cons:

We had some bumps getting ProdPerfect set up for our systems, but they were with us through the entire process to help us get it up a running.