There is an old saying that reads “information is power”, and nowadays this saying becomes more realistic. The way the market moves is based on the demand-offer balance. However, today offers tend to be more accurate than a few years ago and this is because companies compile tons of data to analyze and determine what the next big thing will be.
According to NewVantage Venture Partners, Big Data is delivering the most value to enterprises by decreasing expenses (49.2%) and creating new avenues for innovation and disruption (44.3%).
With that being said, one of the focal areas these days and in the coming future for QA is Big Data Testing. As data itself is to be processed, the challenge lies in creating test scripts, evaluating and providing insights with the tools for Big Data.
Big picture
The big picture of Big Data Testing is based on three main pillars:
1- Validate the input. In a nutshell, this means taking all of the data from very different resources that comes with the 3 V’s:
- Volume - large scale of data,
- Velocity - millions of transactions/data per second
- Variety - multiple sources, multiple formats, etc. and make certain it is well-formed and certify with confidence that once it goes into the processing algorithms of analysis, it will be effectively treated without causing any issue, which, eventually will generate valuable information.
2- Data processing itself: validating the algorithms that are used, which should work from the minimal source of data to the more complex sets of data, that can include both relational and non-relational data. Besides the results coming from those functions and methods, the performance to rapidly handle a lot of information in a very optimized approach.
3- Last but not least pillar is the output validation: are we getting the expected results? This may sound trivial, but in this case, it is not so simple as comparing the income with the outcome. Is more in-depth analysis required?
There is going to be a range of expected information coming out of this cycle of data processing that needs to be consumed by an expert tool and based on the exhaustive analyze.
The QA engineer can make certain it actually can provides value to the customer.
Big Data Testing is nothing like most of the traditional testing activities. It requires a lot of knowledge of the business, of the tools and the expected value coming out of this effort.
Still, the Quality Assurance career is about certifying processes, providing additional value and commitment to quality.
It is also a huge opportunity to learn more about the world we live in, a world in continuous change, and most importantly full of data.
