Smartphones are collecting information about us all of the time. As we post pictures and text online what are we revealing about how we feel, our moods over time?
This is a Brave New World for those who created online services that are designed to substitute for, or enhance, our everyday, in person lives.
A study published in March of 2016 in JAMA Internal Medicine and widely distributed by the media (for example, this article in the San Jose Mercury News) suggests that the creators of these services have a lot of work to do to improve the ability of online personal assistants like Siri and Cortona to recognize and respond to requests for mental health help in a crisis (as opposed to recognizing fashion crises)…
“When asked simple questions about mental health, interpersonal violence and physical health, four widely used conversational agents — Apple’s Siri, Google Now, Samsung’s S Voice and Microsoft’s Cortana — responded inconsistently and incompletely, according to a study published Monday in The Journal of the American Medical Association.”
Now an article from a pair of researchers published online at the MIT website suggests that machines can get pretty good at interpreting our mood based just on what pictures we post on Instagram.
Researchers Andrew Reece at Harvard University in Cambridge, Massachusetts, and Chris Danforth at the University of Vermont in Burlington found significant correlations between the colors in photos posted to Instagram and an individual’s mental health. The link is so strong that the researchers suggested that it could be used to detect early signs of depression… leaving unanswered the larger question of what to do with that information…
As summarized in the MIT article…
The researchers contacted 500 workers from Amazon’s Mechanical Turk service who also had Instagram accounts. They asked these people to complete a series of questionnaires, including a standard clinical depression survey and then they requested that the participants share their Instagram posts for the study.
“Some 170 Turkers agreed, of whom around 70 were clinically depressed. The survey asked various additional questions about their condition, such as the original date of their diagnosis.
The Instagram downloads resulted in a database of over 40,000 photographs which the team then analyzed, again using 
crowdsourcing with a different set of Turkers. For each healthy user, the researchers chose the 100 most recent photographs to be rated. For depressed individuals, the researchers chose the 100 photographs posted before their diagnosis.
These raters were asked to judge how interesting, likeable, happy, and sad each photo seemed on a scale of 0 to 5.
The researchers also evaluated the photographs using objective measures such as the average hue, color saturation, contrast, and so on. This shows how vivid a picture is, for example, and whether it appears gray or faded.
They also counted the number of faces in each image using face 
detection software, on the assumption that faces are a proxy for an individual’s level of social activity. And they assessed the Instagram community’s reaction to each image by counting the number of likes and comments.”
Among other things, the researchers found that those people who were clinically depressed tended to use filters that removed bright colors, tended to post more pictures with faces in them, but that the photos that they posted with faces generally had fewer faces per picture than the photos of people who were not depressed.
Oddly enough, there was no difference between the two groups in the percentage of happy or sad faces posted, nor in the “likeability” of the photos posted.
All told, the algorithm they created correctly identified depression 70% of the time in a second sample of depressed or not depressed individuals.
Ref: arxiv.org/abs/1608.03282 : Instagram Photos Reveal Predictive Markers of Depression