How data lakes can form a culture of innovation

While data lakes are becoming more popular, they should not be a tick-box exercise. Effective use of a data lake can provide insights that make significant improvements to your organizational strategy. But, this can only be achieved with the right resources. Business often talks about having ‘the right people, in the right seats on the bus’, but it’s also about making sure the ride stays interesting.

At this year’s Chief Data Officer Forum Africa, the subject of data lakes was a hot topic, as was retaining top talent. But, while businesses become aware of the advantages of data lakes, many do not understand how to leverage them for maximum results. So although, these may seem like separate topics, they are in fact interconnected.

In the big data environment, many companies are starting to pull away from the traditional approach of Extract Transform Load (ETL) towards Extract Load Transform (ELT). But the emergence of the Data Lake in this shift towards ELT shouldn’t just be viewed as a tick box. In fact, it should be a strategic play to provide your organisation with comprehensive and unique insights to be more competitive.

Once a company appreciates the potential of a data lake, it is in a position to pull incredibly exciting insights. However, to leverage a data lake to its fullest requires a diversity of people and skill sets. In this sense, a culture of innovation that drives continuous learning and collaboration should be considered the ‘best friend’ of exceptional data science.

This is why the topic of data lakes become inextricably linked with attracting and retaining top talent, since extracting insights will be dependent on the capability of the data science team.

Finding a data scientist is tough. In fact, the role is most effectively delivered through two roles, a data engineer and a data analyst. Where a data engineer can populate the data lake and perform the ELT operations, the data analyst can communicate the results to the business in such a way that strategic financial and operational decisions can be made. Consequently, these two roles require different stimulation to sustain high productivity, continuous innovation and ultimately, ensure retention.

Retention therefore, is about making sure the ‘ride stays interesting’. Data engineers and analysts are working in a highly evolving industry so the culture they operate in needs to support this. It requires a strong focus on innovating within roles and structures, which is why the best fit for this industry are environments that create a culture of innovation.

While big business is under constant pressure to sweat its assets, a consultancy firm has more flexibility to invest in retention strategies such as continual learning and a flexible working environment. Consultancies know that best practices are what differentiate them from their competitors, so these are investments they are willing to make. Similarly, big businesses can be assured of a data science team that is constantly stimulated and in-touch with best practice.

Some ways that outsourced consultancies create a culture of innovation:

1. Consulting companies pride themselves in investing heavily in in-house training programmes that ensure clients are impressed with the skills provided. This serves the dual purpose of improving delivery and keeping the staff continually growing and learning. Staff should be incentivised to attend conferences, get involved in the development community.

2. Regarding the work environment for data engineers and analysts, flexibility and freedom will encourage and incentivise the team to come up with new ideas. Providing them with spare time to implement these, whether it be playing around with facial recognition software or thinking of different ways to analyse tweet streams for certain topics, is key to creating a flexible environment.

3. Outsourcing is also beneficial to leverage the strengths of the skills effectively. In build phase, a consulting firm can allocate someone who is strong at ELT’s, mainly setting up and optimising the data into the data lake. Once it’s in the data lake, you can bring on the analysts. This ‘roll-on, roll-off’ approach ensures the client always has the best person for the job and means consultancies can be as agile as the client requires.

4. Lastly, knowledge transference is critical, because support and maintenance tasks are not for everyone. Transferring the IP back to clients by including them in training sessions, and walking the journey together to build up the internal capacity, means consultancies and clients can both get the most out of the partnership.

The world of Big Data is changing quickly. I believe that if you are looking to select a partner to accompany you on the journey of implementing a meaningful big data solution, you need to find a company with a culture of innovation that is always looking for new ways to do things, learning new technologies, and staying up-to-date with trends. While the inclusion of a data lake may provide great opportunities for business, a data science team that can adapt to industry culture and remain relevant will keep the ride interesting and provide you with the best results.