Welcome!

Cloud Security Authors: Yeshim Deniz, Liz McMillan, Elizabeth White, Ravi Rajamiyer, Pat Romanski

Related Topics: @DXWorldExpo, @CloudExpo, Cloud Security

@DXWorldExpo: Blog Feed Post

Big Data and the Personalization Challenge: The Murky Middle By @Schmarzo | @BigDataExpo #BigData

Personalized recommendations are a real dilemma in the world of big data

What if you knew, through legal means, something about someone where you could intervene to deliver advice to help them perform better or stay out of danger? Should you act on that?

I was recently delivering a lecture at a major university in Texas and one of the participants posed a couple of very thought-provoking scenarios (stay with me as these are sort of long, abstract scenarios, but hey, what better way to provoke an interesting conversation):

What if by virtue of monitoring a student’s social media habits (with the student opting to share their social media activities), the university could 1) flag behaviors that predict student classroom performance problems and 2) could deliver recommendations to the student, faculty and advisors for improving the student’s social, classroom and study habits and increase their probability of graduating on time and with higher grades?

What if by monitoring the student’s study habits and extra-curricular activities (e.g., how often they went to the library and/or the lab and for how long, class attendance, attendance at campus events, extracurricular activities, etc.), the university could generate recommendations to the student, faculty and advisors for improving the student’s social, classroom and study habits and increase their probability of graduating on time and with higher grades?

These scenarios raise all sorts of ethical questions, such as:

  • Would not this be in the best interests of the students to perform this analysis in order to help the students achieve their best outcomes?
  • Would students want this level of analysis as students come to college to earn a degree and achieve a level of classroom performance that will help them get better jobs and earn more money?
  • Is one type of monitoring more intrusive and a potential violation of the student’s privacy (even if they have opted in to share their social media activities)?
  • Where does one draw the line between what is helping the student to increase the probability of their college and post-college success, and what is being a “creeper”?

One of the workshop participants called this the “murky middle” where noble and forthright efforts to help students to be more successful may cross the personal privacy line.

Primary Research (My Kids as Guinea Pigs)
I’ve got to be honest. I don’t have an answer. And I’m really torn by the dilemma of wanting the best performance for students (and my kids), but doing it in such in a way that doesn’t violate their personal privacies or come across as being a “creepy.”

So I asked my children what they thought of these scenarios. Each of my children has a unique perspective:

  • Alec has already graduated from college and has been in the workforce for a couple of years now.
  • Max is currently in grad school and will be living these two scenarios over the next few years.
  • Amelia is evaluating colleges and has to determine if she is comfortable with these levels of monitoring.

Here are their answers:

Scenario #1: What if by virtue of monitoring a student’s social media habits (with the student opting to share their social media activities), the university could 1) flag behaviors that predict student classroom performance problems and 2) could deliver recommendations to the student, faculty and advisors for improving the student’s social, classroom and study habits and increase their probability of graduating on time and with higher grades?

Amelia (evaluating colleges): If I gave my social media information to my future college, I would expect them to skim over my social media accounts. I think in the beginning it could be a great benefit, but for an entire year I do not think it would be beneficial. I believe this because everyone wants to be successful in school and life, but a constant watch over the accounts may get stressful and not necessary. It can often create frustration and lead to hidden activities that could result in something more detrimental.

Max (still in college): If there were a human component to it, I would feel like the college is investing in me. However, if it were electronic I would feel violated. For example, if my college had access to all of this information and a counselor gave me a call, I would feel like someone cares. But if I got an email or text message on my phone offering me suggestions I would feel violated and “big brothered.”

Alec (out of college): I don’t like the idea of anyone – especially a school or other authority type institution – examining my social media to identify trends in terms of how I “tick.” There is something decidedly unsettling about any entity, for whatever purpose, altruistic or not, analyzing my social relations and providing advice on how to improve myself. Ditch certain friends? Don’t go to certain parties? These types of recommendations would only lead to backlash to the system as a whole.

Scenario #2: What if by monitoring the student’s study habits and extra-curricular activities (e.g., how often they went to the library and/or the lab and for how long, class attendance, attendance at campus events, extracurricular activities, etc.) to generate the recommendations to the student, faculty and advisors for improving the student’s social, classroom and study habits that will increase their probability of graduating on time and with higher grades?

Amelia (evaluating colleges): Similar to the first scenario, this scenario would only be helpful in the beginning. The main purpose to going to college is to receive an education on my own. Constantly being tracked and told where to go would help you in college, but would not set you up for the real world. In the real world, you are not going to have someone tell you to go study more or go to classes more. If you miss a class or don’t study, you must face the consequence. Overall, growing up and making mistakes is a part of life and to have that stripped at such a young age may lead to a hard reality check.

Max (still in college): At no point in time would I be okay with my college knowing my location. I feel that crosses the line of caring. I wouldn’t give my college access to something I wouldn’t give my parents access to. To be honest dad, I wouldn’t feel comfortable if you knew how many classes I missed and whether or not I was at the library. I am sure the government knows, but sometimes with electronics, ignorance is bliss.

Alec (out of college): This option is more feasible in my opinion. I would be much more comfortable with an educational institution analyzing my educational decisions. Analysis of my study habits, correlations in terms of grades and time spent in the library or class attendance would prove invaluable in making a logical case to present to the student to convince them that their grades are affected by these factors.

So none of my kids would be happy with this level of monitoring and analysis, with some specific observations:

  • While Amelia (evaluating colleges) felt that some level of monitoring and analysis might be useful at first, it would eventually impede her personal development. Students are going to college to learn to live on their own; monitoring their behaviors and “holding their hands” doesn’t allow them to grow.
  • Alec (already graduated) seems comfortable with the school using some of the classroom data to make performance improvement recommendations, but neither Max (still in school) nor Amelia (evaluating schools) feel comfortable with this.

However, if colleges were to do this level of monitoring and analysis, then the advice or recommendations must have a human component. By having a counselor get involved, you give the data a human face.

I think this last point is critical – give the data a human face. As I’ve discussed many times, the real impact of big data (and data science) is helping the humans in the process by providing recommendations to the humans (teachers, physicians, parole officers, technicians, counselors, coaches, therapists) to make them more effective at the point of customer engagement.

Summary
Let me give you one more scenario to chew on: What if by virtue of analyzing the student’s social media and classroom data, the college is able to make the following probabilistic determinations:

  • There is 65% chance of the student dropping out or flunking out
  • There is a 33% chance that the student is having substance abuse problems
  • There is a 5% chance that the student is going to attempt suicide

What should the college do?

Personalized recommendations are a real dilemma in the world of big data – how much monitoring and analysis is “okay” before crossing over that fine line. And that’s the heart of the challenge: it isn’t a fine line; it’s the murky middle.

Big Data and the Personalization Challenge: The Murky Middle

Read the original blog entry...

More Stories By William Schmarzo

Bill Schmarzo, author of “Big Data: Understanding How Data Powers Big Business” and “Big Data MBA: Driving Business Strategies with Data Science”, is responsible for setting strategy and defining the Big Data service offerings for Hitachi Vantara as CTO, IoT and Analytics.

Previously, as a CTO within Dell EMC’s 2,000+ person consulting organization, he works with organizations to identify where and how to start their big data journeys. He’s written white papers, is an avid blogger and is a frequent speaker on the use of Big Data and data science to power an organization’s key business initiatives. He is a University of San Francisco School of Management (SOM) Executive Fellow where he teaches the “Big Data MBA” course. Bill also just completed a research paper on “Determining The Economic Value of Data”. Onalytica recently ranked Bill as #4 Big Data Influencer worldwide.

Bill has over three decades of experience in data warehousing, BI and analytics. Bill authored the Vision Workshop methodology that links an organization’s strategic business initiatives with their supporting data and analytic requirements. Bill serves on the City of San Jose’s Technology Innovation Board, and on the faculties of The Data Warehouse Institute and Strata.

Previously, Bill was vice president of Analytics at Yahoo where he was responsible for the development of Yahoo’s Advertiser and Website analytics products, including the delivery of “actionable insights” through a holistic user experience. Before that, Bill oversaw the Analytic Applications business unit at Business Objects, including the development, marketing and sales of their industry-defining analytic applications.

Bill holds a Masters Business Administration from University of Iowa and a Bachelor of Science degree in Mathematics, Computer Science and Business Administration from Coe College.

IoT & Smart Cities Stories
The deluge of IoT sensor data collected from connected devices and the powerful AI required to make that data actionable are giving rise to a hybrid ecosystem in which cloud, on-prem and edge processes become interweaved. Attendees will learn how emerging composable infrastructure solutions deliver the adaptive architecture needed to manage this new data reality. Machine learning algorithms can better anticipate data storms and automate resources to support surges, including fully scalable GPU-c...
Machine learning has taken residence at our cities' cores and now we can finally have "smart cities." Cities are a collection of buildings made to provide the structure and safety necessary for people to function, create and survive. Buildings are a pool of ever-changing performance data from large automated systems such as heating and cooling to the people that live and work within them. Through machine learning, buildings can optimize performance, reduce costs, and improve occupant comfort by ...
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
Predicting the future has never been more challenging - not because of the lack of data but because of the flood of ungoverned and risk laden information. Microsoft states that 2.5 exabytes of data are created every day. Expectations and reliance on data are being pushed to the limits, as demands around hybrid options continue to grow.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
As IoT continues to increase momentum, so does the associated risk. Secure Device Lifecycle Management (DLM) is ranked as one of the most important technology areas of IoT. Driving this trend is the realization that secure support for IoT devices provides companies the ability to deliver high-quality, reliable, secure offerings faster, create new revenue streams, and reduce support costs, all while building a competitive advantage in their markets. In this session, we will use customer use cases...