Installing R and the package igraph on a Mac: As Always, Not Quite the Same

The incredibly useful research program called R is available on many platforms — Linux, Windows and Apple computers — and can run the same scripts across all three of its different versions.  That said, the experience of getting R to run those scripts is not quite the same on an Apple Mac.  This seems to be some kind of unwritten rule for Macs — whatever your program, on a Mac the menus, procedures and names of commands will somehow end up being different.

So what?  Well, if you’re just getting started with R, you’ll need to occasionally get some tips and tricks for making the program work.  Most of the how-to blog posts and videos you can find out there use examples using a Linux or Windows system — and they just won’t work for a Mac.  I found this out the hard way when teaching students to use the igraph package for R to perform social network analysis.  A few of my students have Macs at home, and it didn’t take long for them to cry for help, because the R program they were dealing with looked very little like the R program I’d been showing them.

If you find yourself in the same boat, and are running into trouble using R and igraph, I hope the following video will be of some help. Using a screen capture of a Mac running OS X, I briefly demonstrate the experience of installing R and running a script with the igraph package on from an Apple vantage point.  One difference is that there are a few menu options you’ll need to select when installing igraph to actually make it run.  In another simple but crucial difference for Macs, you’ll need to select all the text in your script before running it.  THEN, and only then, use the “Execute” command.  That’s not necessary on a Windows computer, but it’s a make-or-break move on a Mac.

Why? Don’t ask me why. It’s the same old story that we’ve had for thirty years: it’s just different on a Mac.

The walkthrough video:

Please leave a comment if you have a question or need clarification, and I’d be glad to be of help if I can.

Social Media Data Mining with Raspberry Pi: 8 Videos for the Complete Beginner

Since the start of this year, I’ve been working on a project to take a $30 Raspberry Pi 2 computer turn it to create a social media data mining machine using the programming language Python. The words “programming language” may be off-putting, but my goal is to work through the process step-by-step so that even a complete beginner can follow along and accomplish the feat.

The inexpensive, adaptable $30 Raspberry Pi 2I’m motivated by two impulses. My first impulse to help people gain control over and ownership of the information regarding interaction that surrounds us. My second impulse is to demonstrate that mastery of social media information is not limited to the corporate, the government, or the otherwise well-funded sphere. This is not a video series for those who already are technologically wealthy and adept. It’s for anyone who has $30 to spare, a willingness to tinker, but the feeling that they’ve been left out of the social media data race. I hope to make the point that anyone can use social media data mining to find out who’s talking to whom. The powers that be are already watching down at us: my hope is that we little folks can start to watch up.

I’m starting the project by shooting videos. The video series is not complete yet (I have yet to film videos on automation using cron jobs and input statements), but has proceeded far enough along to represent a fairly good arc of skill development. Eventually I’d like to transcribe the videos and create a written and illustrated how-to pamphlet; these videos are just the start.

Throughout the videos, I’ve tried not to cover up the temporary mistakes, detours and puzzling bugs that are typical of programming. No one I know of hooks up the perfect computer system or writes a perfect program on the first try. Working through error messages and sleuthing through them is part of the process, and you’ll see that occasionally in these videos.

Please feel free to share the videos if you find them useful. I’d also appreciate any feedback you might have to offer.

Video 1: Hardware Setup for the Raspberry Pi

Video 2: Setting up the Raspberry Pi’s Raspbian Operating System

Video 3: Using the Raspberry Pi’s Text and Graphical Operating Systems

Video 4: Installing R

Video 5: Twitter, Tweepy and Python

Video 6: Debugging

Video 7: Saving Twitter Posts in a CSV File

Video 8: Extracting and Saving Data on Twitter URLs, Hashtags, and Mentions

2015 American Community Survey Table: U.S. Immigrants are Less Likely to be in Adult Corrections Facilities than those born in the U.S.A.

2015 American Community Survey: Immigrants Less Likely to be Housed in Adult Corrections Facilities

Every September, the U.S. Census Bureau releases data regarding the U.S. population from its annual American Community Survey. The American Factfinder website very handily archives this data and makes it available through guided or customized search.

I particularly encourage you to visit American Factfinder and search for a table titled “CHARACTERISTICS OF THE GROUP QUARTERS POPULATION BY GROUP QUARTERS TYPE.” That table sounds dry and uninteresting, but it contains a nugget of gold for any voter who wants to fact-check claims being made lately about immigrants.  In press releases and in speeches this year, political officeholders and candidates have asserted that immigrants to the United States are dangerous and liable to commit crimes.  Of course, it is possible to find tragic stories of crimes committed by immigrants to the United States, just as it is possible to find tragic stories of crimes committed by people born in the United States.  But individual stories are not a good basis for policy. Claims about immigrants as a source of crime are strong in their accusation and as such need to be evaluated on the basis of systematic evidence.

To cut to the chase, data from this table reveal that immigrants make up a lower share of people held in adult corrections facilities in the United States than their share of the U.S. population.  “Native born” Americans — those born in the United States — made up 86.5% of the U.S. population in the 2015, but made up 91.9% of those housed in adult correctional facilities in the United States in 2015.  The “foreign born” immigrants to the United States made up 13.5% of the U.S. population in 2015, but made up only 8.1% of those housed in adult correctional facilities in the U.S. in 2015:

2015 American Community Survey Table: U.S. Immigrants are Less Likely to be in Adult Corrections Facilities than those born in the U.S.A.

This data does not appear to be consistent with the claim that foreigners coming to the United States to live are a unique and concentrated source of crime.  Trends for 2015 match findings for previous years compiled for the National Bureau of Economic Research.  Those who wish to pursue policies against immigrants on the basis that doing so would cut crime rates in the United States need to explain how their assertions match these observations.

Social Media Accounts of Candidates for the Maine State Senate

Deciding who to vote for in state legislative campaigns can sometimes be tricky because thorough coverage of local candidates can be hard to find. In the state of Maine,  state legislators in Maine are known for their accessibility. This may be because Maine’s legislative districts tend to be small; it may also be due to the friendly nature of Maine folk in general. Whatever the reason, getting in touch with candidates for Maine political office is both important and possible.

In this day and age, the quickest way to learn about state legislative candidates and to find their contact information is through social media platforms like individual web pages, Facebook and Twitter.  To help you in that process, the I’ve put together a spreadsheet with information about the social media presence of the 70 candidates for the Maine Senate in 2016, along with some additional contextual information. To download this information for personal use, click here for a Microsoft Excel file.

This sort of information changes all the time — if you have updated information about new accounts, please share a comment below to let me know, or write to james.m.cook@maine.edu.

Two Fact Checks on Donald Trump and Crime

In tonight’s speech, Donald Trump will accept the presidential nomination of the Republican party. The text of Trump’s speech makes the following claims regarding crime in the United States:

“These are the facts:

“Decades of progress made in bringing down crime are now being reversed by this Administration’s rollback of criminal enforcement.

“Homicides last year increased by 17% in America’s fifty largest cities. That’s the largest increase in 25 years.”

Let’s look at these two claims and check the facts.

Fact Check of Claim 1: “Decades of progress made in bringing down crime are now being reversed by this Administration’s rollback of criminal enforcement.”

Response: The annual FBI report Crime in the United States provides the most recent data on crime, both in the United States overall and in particular communities. Annual reports are released every fall to describe crime in the year before, based on direct reports of police officers all over the country (the delay occurs because it takes time to gather all those reports and carefully tabulate them). The most recent report was released in 2015, describing crime in the year 2014. Anyone who tells you they know about U.S. national crime trends for any more recent year is fibbing — because 2016 isn’t over yet, and because final counts for 2015 are still being worked on.

The trends on violent and property crime victimization rates in the United States are shown below, from the very first page of the 2015 Crime in the United States report, released at the end of September 2015:

violent and property victimization rates in the United States from 1993 to 2014

People can disagree about policy, but it is not possible for policy changes to have led to a reversal in progress in the crime rate in the United States, because there is no evidence that such a reversal exists.

Fact Check of Claim 2: “Homicides last year increased by 17% in America’s fifty largest cities. That’s the largest increase in 25 years.”

We don’t actually know whether this is the case, because final data for 2015 is not yet available. A preliminary count, that is not a final count, that is only for the first six months of 2015, and that is only for cities with over 100,000 in population — has been released. Here it is. Let’s realize, based on this data (look at Table 4), that:

First, we do not yet actually have a final count for 2015.

Second, on the basis that only the first six months of 2015 have been counted, it is not possible to make the conclusion that Donald Trump makes regarding the entire year.

Third, if we actually look at the fifty largest cities in the United States, and look at the preliminary count for the first six months of 2015 (not the entire year), we find that the homicides are up 8.4% in America’s fiftiest largest cities compared to 2014 — Donald Trump’s speech claims twice as much as this.

Fourth, it’s interesting that the speech only focuses on homicide, and not on violent crime in general. The increase in the violent crime rate from 2014 to 2015 is 3.1%.

Fifth, even these rises do not take into account the rise in population of America’s fifty largest cities, increasing the population, which will of course increase the number of murders.

Sixth, even this increase, in the context of the huge falls of the last twenty years, still marks a low crime rate in America’s fifty largest cities in recent history. The preliminary homicide rate in the fifty largest cities of the United States in the first six months of 2015 was 4.06 homicides per 100,000 people. In the first six months of 2015, the overall violent crime rate was 305.7 per 100,000 people. By comparison, in 2008, when Barack Obama was elected president and the decline in homicides was already well underway, the homicide rate was 12.1 per 100,000 people in America’s 50 largest cities, and the overall violent crime rate was per 852.9 per 100,000 people. In other words, since Barack Obama became president, if the 2015 preliminary data holds, the homicide rate is down 66.4% and the violent crime rate is down 64.2%.

It turns out that Donald Trump’s claim is based on a post made in very early estimate by a blogger using very early data in January 2016, less than a month after 2015 ended.

This second claim by Republican nominee Donald Trump, like the first, is not supported by the facts.

Another Season, Another Public Figure Self-Immolates with Plagiarism

Another season, another new public plagiarism case.

It’s not a partisan thing. A different season, a different public plagiarism case:

David Greenberg: Why Biden’s plagiarism shouldn’t be forgotten.”

Students, listen up: Of course there are moral implications when you steal others’ words and pass them off as your own. Of course the choice to plagiarize keeps you shallow because you haven’t bothered to do the work of thinking for yourself. Even if you don’t care about that, plagiarism is a horrible strategic choice. Your reputation will be destroyed. You’ll find yourself making the defense that you’re not malicious, “just” incompetent and sloppy. Any actual original work you do will be discounted. Your career will be stunted.

The best protection against plagiarism is to use your own brain to think up, then write, something truly original and your own. If you can’t manage that, then why are you writing or speaking in the first place? For written forms that require thorough research of others’ contributions — to which you then should add your own original thoughts — be polite: quote and cite. Although academic integrity policies can take many words to express the standard, avoiding plagiarism really is that simple.

A tip of the hat to journalist Jarrett Hill for uncovering the latest in intellectual theft.

Presentation Materials for Twitter Adoption in U.S. Legislatures at #SMSociety 2016 Conference

The following are links to supporting materials for the presentation “Twitter Adoption in U.S. Legislatures: A Fifty-State Study” made to the 2016 International Conference on Social Media & Society on Wednesday, July 13 at Goldsmiths, University of London.

1. Free full-text access:

ACM DL Author-ize serviceTwitter Adoption in U.S. Legislatures: A Fifty-State Study

James M. Cook
SMSociety ’16 Proceedings of the 7th 2016 International Conference on Social Media & Society, 2016

2. Download Powerpoint Presentation Slides from presentation

3. Abstract: This study draws theoretical inspiration from the literature on Twitter adoption and Twitter activity in United States legislatures, applying predictions from those limited studies to all 7,378 politicians serving across 50 American state legislatures in the fall of 2015. Tests of bivariate association carried out for individual states lead to widely varying results, indicating an underlying diversity of legislative environments. However, a pooled multivariate analysis for all 50 states indicates that the number of constituents per legislator, district youth, district level of educational attainment, legislative professionalism, being a woman, sitting in the upper chamber, holding a leadership position, and legislative inexperience are all significantly and positively associated with Twitter adoption and Twitter activity. Controlling for these factors, legislator party, majority status, partisan instability, district income, and the percent of households in a state with an Internet connection are not significantly related to either Twitter adoption or recent Twitter use. A significant share of variation in social media adoption by legislators remains unexplained, leaving considerable room for further theoretical development and the development of contingent historical accounts.

Please feel free to review these materials before or after my presentation. I look forward to your comments.

Social Limits are Real: they can End (or Save) your Life

(cross-posted in revised form from an ongoing column for the Kennebec Journal)

Part I: No Limits?

As part of my work in social media research, I follow the Twitter accounts of Maine’s elected officials, studying patterns in their public communications.

I wasn’t surprised to see graduation messages from Rep. Justin Chenette, D-Saco, in my feed this month; ’tis the season, after all, and local officials are regularly invited to speak.Maine State Representative Justin Chenette writes, "The only limitations, are the ones we set for ourselves

“The only limitations, are the ones we set for ourselves.”

“You can achieve anything you set your mind to.”

Ideas like these are commonly asserted: during the first half of the month of June 2016, the phrase “no limits” was posted to Twitter at a rate of more than 100 times per hour. That’s understandable.  The idea of no limits in life, of a plain of possibilities open wide before us, is a hopeful idea. It’s an inspiring idea.  It’s a popular idea.

Unfortunately, it’s also an incorrect idea.

Don’t believe me? Allow me to issue two challenges. If there really are no limits in life, try:

• Heading to Baxter State Park here Maine, then walking straight through the base of Mount Katahdin.

• Jumping so high into the sky that you launch yourself into orbit, without assistive technology.

No matter how much you wish, no matter how hard you try, you can’t accomplish these acts. We are all limited by laws of physics. Due to electromagnetic repulsion, you can’t walk through rock. Your legs just can’t generate enough kinetic energy to overcome the force of Earth’s gravity. Ignoring these limits can kill you.

Just as there are laws of physics, there are social laws that set limits in society. Consider the following image of a social network, in which circles represent people and lines represent relationships between them:

A spcial network of minority blues and majority reds

As you can see, there are two groups of people in this network: reds and blues. Sociologist Peter Blau considered the rate of “outgroup association” (the percent of all relationships by group members to people outside the group) and discovered a law: that smaller groups have a higher rate of outgroup association than larger groups.

The reds in the network have five relationships, two of which are to non-reds. 2/5 = 60 percent. The blues in the network have 10 relationships, two of which are to non-blues. 2/10 = 20 percent.

Go ahead, add and subtract as many ties from this network as you wish. You can’t make the pattern go away. It’s an inescapable law.

So what? Why does this social law matter? Stop thinking about reds and blues and start thinking about numerical minorities: women in male-dominated corporate boardrooms, immigrants in native-born communities, or black people in majority-white America. Due to math alone, no matter how much you wish it to be otherwise, such minorities will have contact with majority members more often than majorities have contact with minorities.

Because of this pattern, minority groups must devote more energy to knowing the habits of two cultures just to get by. The stakes for failure can be high. A woman who ignores male culture in the boardroom can lose a job. An immigrant who can’t speak the language loses business. When northerner Emmett Till whistled at a white woman in the South, he lost his life.

This is one social law, one inescapable limit among many in society. A sense of limitless potential may feel good to this year’s crop of graduates, but if that feeling is untempered by a reality check, the outcomes can be disastrous.

Part 2: Deadly Friendships

Sociology’s laws are subtler than the laws of physics, but they’re no less deadly.

For another example of sociological laws and their consequences, let’s consider Feld’s Law. Twenty-five years ago, sociologist Scott Feld demonstrated that on average, your friends have more friends than you do. This sounds impossible, but it’s true.

To show you what I mean, we’ll look at another social network.  In this network, circles indicate people and lines indicate friendships between them:

A simple social network used to demonstrate Scott Feld's maxim regarding friendship among friends.

Some people in this network, like Carol, have more friends than others, like Hal.  Popularity and unpopularity is a normal part of human social life.

Also, some members of this network are friends with very friendly people; Don’s friends have 3.5 friends of their own on average (Carol with 4 friends, Ed with 3). On the other hand, the average friend of Carol has just 2.25 friends (Al, Betty and Don with 2 friends each, and Gina with 4).  That sort of variation shouldn’t be too surprising either.  But as Feld found, things get strange when we consider the overall trend.

Take a look at the table below, which shows the results for each person as well as the overall average for all people. The average person in our network has 2.5 friends. The average person’s average friend has 2.9 friends. That’s exactly the sort of result that Feld’s law predicts.

A table adding up friends, friends of friends, and averages for each of these in a social network.

Go ahead, draw your own social networks and do the math for yourself. You’ll find that Feld’s law holds true for almost any network you can think of. That’s odd, but why should you care?

One reason to care is that Feld’s law explains that feeling many of us have that we’re less popular than our friends. Odds are, you’re right. Don’t take it personally; it’s just the way societies work.

Another reason to care is that Feld’s law is also true for any relationship in which people share something with one another. People share needles in the opiate epidemic facing not just the state of Maine where I work but significant swaths of the North American continent more broadly. People have shared sexual relations since the dawn of humanity. These kinds of sharing can also share deadly viruses like AIDS, hepatitis B and syphilis.

If you’re thinking of sharing a needle, Feld’s law tells us that on average, the people you share needles with share needles with more people than you do. Feld’s law tells us that on average, the people with which you have unprotected sex have unprotected sex with more people than you do. This means that even if you share a needle just once, even if you have unprotected sex just once, your chances of catching a killer virus are disproportionately high.

Understanding invisible social laws are crucial for getting by in life because they provide guides for safe and unsafe behavior, just as physical laws guide us while we climb a mountain or skirt a rooftop. When you support research to uncover sociological laws, the benefits aren’t abstract. Understanding how society works can save a life — and that life could someday be your own.

A Hashtag Crash: CCS2016, Meet CCS2016, CCS2016, CCS2016 and CCS2016

Visit the Twitter hashtag channel #CCS2016 for information on the 2016 Conference on Complex Systems taking place in Amsterdam this upcoming September. Well actually, isn’t #CCS2016 the hashtag covering the 2016 Canadian Crowdfunding Summit? Or, wait, does #CCS2016 refer to the 2016 Content and Commerce Summit meeting in Orlando, Florida? Or is #CCS2016 the hashtag for announcements regarding 2016 Comic Con Spain? Could #CCS2016 be a hashtag for a cinematography conference in Caracas, Venezuela?

The answer is yes, yes, yes, yes, and yes. The Twitter hashtag channel #CCS2016 has been used to promote all of these, a simultaneous indication of the popularity, bottom-up flexibility, and strategic difficulty involved in using the social media platform. #CCS2016 has been used beyond this, within the past year referring to events as diverse as a Brazilian country music festival, a high school spirit effort, a “Corporate Community Summit” and an academic conference on “Cities as Community Spaces.”

The graph below features all participants in the #CCS2016 hashtag channel from April 1 to 11, 2016. Every dot (called a node) represents a Twitter account that has made a post including the #CCS2016 hashtag. Every line (called a tie) represents an instance in which one Twitter account has mentioned or replied to another Twitter account. Together, these nodes and ties make up a springtime social network for CCS2016.

Social Network for Twitter accounts using the #CCS2016 hashtag from April 1-11, 2016

As you can see, this is a disconnected network.  The large dark blue network at the top of the graph consists of Twitter users discussing Comic Con Spain.  The smaller light blue network below it is beginning to grow after the announcement of an annual Conference on Complex Systems.  To its left are participants in the upcoming Canadian Crowdfunding Summit.  To the lower right are a handful of remaining nodes discussing less popular or timely representations of the title “CCS2016.”

Separate conversations are put in separate areas of this two-dimensional graph.  On Twitter itself, no such separation is afforded. The purpose of a hashtag is to provide a space that community members can visit when they have something to say, or have a desire to listen.  In this busy, muddied virtual room called “#CCS2016,” multiple conversations are taking place on top of one another.

Why don’t all these different groups use a different hashtag?  A social media marketer would advise always reviewing past use of a phrase before adopting it as a hashtag of one’s own, lest one be accused of acting as a hashtag crasher.  But regardless, lines of distinction in Twitter can help keep the conversation coherent.  Different users are speaking different languages: English, Portugese, Spanish.  This is a sorting mechanism.  A second sorting mechanism comes from the ties charted in the graph above: people with a particular interest in a hashtag are most likely to find out about that hashtag because they follow other people with the same particular interest.  This means relevant hashtag posts are most likely to appear in a user’s Twitter timeline.  Finally, popularity of particular uses for a hashtag may shift over time, as one event comes to a head and another recedes into the past.

At times, it’s a mess, but this is what civic democracy looks like.

1 2 3 6