I’d been waiting upwards of 10 minutes to return a purchase for my mom. Waiting patiently at the Walmart customer service desk while the employee behind the desk took a call from someone who had left something in the store. They talked, the employee wrote the missing item on a scrap of paper (I couldn’t see what it was), and then hung up and put the paper in the binder.
When the employee finally helped me, I couldn’t help but think, Wow. Even with the storage and retrieval sophistication in software and computer systems today, we’re still putting scraps of paper in a binder, never to see the florescent lighting of the store again.
Crazy I know. Now imagine over 10,000 lost and found binders at all the Walmarts worldwide.
You know what’s even crazier? According to IBM, human beings create 2.5 quintillion bytes of data. Every day.
Did you get that? That’s 2,500,000,000,000,000,000 bytes of data. I’m just excited that my oldest daughter, who’s four, can count to 20.
Oh, and by the way, that volume of data is just an average over the past two years. Can you imagine the next two years? For those of you keeping score at home, big data in the IT world is a collection of data sets so large and complex that it becomes difficult to process using smaller database management tools. It takes pretty powerful software, hardware processing power, and multiple servers to process it all, which we have today. In fact, software startups that are crunching somewhat large amounts of data for their customers can quite affordably lease the computer power and server storage needed to do so. Barriers to entry in the big data world are smaller than they’ve ever been.
Today big data is everywhere: Internet search, finance and business information, HR and recruiting information, biological research, meteorology, physics, environmental research, and much more. It’s also being collected from all over the place, everywhere: climate sensors, mobile devices, remote sensing, software logs, social media posts, digital pictures and videos, microphones, radio-frequency identification readers, purchase transaction records, and cell phone GPS signals. Just to name a few.
2,500,000,000,000,000,000 bytes of data. Every day.
According to DOMO, a business intelligence company, the amount of data generated on the Internet every minute by a global Internet population of 2.1 billion people includes:
- 48 hours of video on YouTube
- 684,478 pieces of content shared on Facebook
- 3,600 new photos shared on Instagram
- 27,778 new posts published on Tumblr
I’ve seen other, similar daunting stats. I mean, how do we make sense of all this? Maybe all this “big data” doesn’t rock you to your very core as it does me, but you’ve got to admit it’s a little unsettling. And even a little silly. Don’t you think?
A little silly when you put millions of scraps of paper in a binder. No one’s finding anything. Ever. What we need is more help finding, aggregating, analyzing and delivering the data in a way that this Daddy of two can understand. One and two. That’s all you’re getting from me.
I’m telling you, there are lots of big data jobs out there. But as with all constantly changing tech jobs today, you’ve got to have the right skills and some contextual experience.
According to management consulting firm McKinsey & Company, the United States could be in a pickle by 2018, with a shortage of up to 190,000 people with “deep analytical talent” and upwards of 1.5 million people capable of analyzing data that would enable business decisions.
Based on the research I’ve done, there are lots of jobs available. But there’s just not a definitive “big data” profession; the skill sets required reach across almost every business and industry. Just search for “big data” jobs using your favorite job search tool. There are thousands and that number will only grow.
Tech experts agree that math, statistics, data analysis, business analytics, natural language processing, creativity, communications skills, and Hadoop are all critical to creating big data analytics frameworks and applying them to benefit businesses. Hadoop is an open-source software framework that supports tons of data and enables multiple applications and computers to work together. Hadoop can be learned via classes or can be self-taught.
Again, because there’s not a “big data profession” per se, here’s a list of the most common jobs with the skill sets required based on recent Computerworld research:
Data Scientists: Today, these are the smartest folks in the room with the deepest of analytical insight, who are also assuming leadership positions in businesses. Lots of math and statistics here as well as artificial intelligence, natural language processing, and/or data management.
Data Architects: These are programmers who can put the method to the madness when it comes to that mountain of disconnected data. They usually have statistical backgrounds as well as programming and BI, and have the ability to look at data in new and innovative ways.
Data Visualizers: These are the folks who interpret the data and analytics for everyday business use. They give the data context and understandability for all in business to consume and learn from and to apply to daily operations.
Data Change Agents: These are the quality management and change management folks who use the data analytics to improve operations and processes. They are clear communicators and help to ensure the data analyses stick. Data Engineer/Operators: These are the folks who take the big data building blocks and create the infrastructure and manage it, ensuring that business needs are met and systems and processes are performing as they should.
And here’s a “Mindful Moment” for you: The sheer volume of data in the world today is unfathomable. The number of zeros alone at the end of the bytes of data pumping through the our daily veins is enough to make me pass out multiple times. But the good news is that all this data of the democratization of the Internet is creating a whole bunch of “big data” jobs.
I know I’ve got my binders out and ready to go. And you?
This article includes excerpts from my new book, Tech Job Hunt Handbook, available now for preorder from Apress.