Scrape & Map Tweets

This is a how-to guide to scraping (archiving tweets), saving the tweets to a google spreadsheet, and migrating the data to a Google Fusion Table, and finally producing a map of where the user is located (if they have set that in their profile).

This does not produce a live map of Tweets.

You will need some third party apps:

Google Drive: https://drive.google.com/

Google Sheets (Found on Google Drive)

Google Fusion Tables (Found on Google Drive)

Twitter: http://www.twitter.com

Twitter archiver, found here: https://chrome.google.com/webstore/detail/twitter-... (I will run you through this)

You should have:

A Google Account: https://accounts.google.com/signup

Prerequisites:

A Google Account

Fusion Tables Installed

A Twitter Account

Be familiar with Twitter's Advanced Search: https://support.twitter.com/articles/71577

It is recommended that you have first checked out my Instructables, Create a Clickable Google Map to Share Geographic Information and Metadata. I cover how to create a Google Drive account, how to create a spreadsheet, and how to install Google Fusion Tables: https://www.instructables.com/id/Create-a-Clickable...

Twitter Archiver lets you easily save tweets for any search keyword or hashtag in a Google Spreadsheet. Enter a search query, or an hashtag, and all the matching tweets are automatically saved in the Google Sheet. You can write simple queries, use boolean search or include advanced Twitter search operators to create more complex queries.

1) Go to Twitter Archiver https://chrome.google.com/webstore/detail/twitter-...

2) Click on "+ Free"

3) You will be asked to sign in, or you if you are signed in, you will be redirected to a new Google Sheet.

4) Twitter Archiver will ask for your permission to run. Click "Continue"

5) Twitter Archiver will ask to change permissions. Click "Allow"

A special note on privacy- As you can see, Twitter Archiver asks you to share a lot of private data. Please take the time to read through the permissions to see if you agree with them.

After installing Twitter Archiver, you will be brought to a blank spreadsheet. This is the Twitter archiver!

1)You will need to authorize Twitter to get started. Click on "Add-ons" at the top, Click "Twitter Archiver" and then click "Authorize Twitter".

2) You will need to connect to the Twitter API. Click the blue "Authorize Twitter" button

3) You will be redirected to Twitter's login page. Sign in with your Twitter account.

4) Once you sign in, a screen will appear that tells you your Google Account is authorized to use Twitter. Close that tab.

You will now create your own Twitter Search Rule.

For information on advanced Twitter searches, check out Twitter's documentation. You can create rules that mention certain search terms, look for exact phrases, find tweets by #hashtags, tweets that @mention particular Twitter users and so on.

Type in your search terms and hit "Start Tracking". The Twitter Archiver will begin doing its job! After a little bit of time, you will have a spreadsheet full of Tweets. You will want to search for a term that will produce a lot of geographic information, so maybe a popular #hashtag will work best.

At any time, you can rename your Spreadsheet. I typically name mine "Twitter Archiver", as I imagine it running as a program or app with that name.

Note on the results of a search: The Twitter API does not provide all the historical tweets via the API. It will only return tweets that have been posted in the previous 5-7 days or last 1000 tweets. The archiver will however fetch future tweets for any terms that it is tracking. The free edition of the Twitter Archiver lets you track up to 2 search queries. The free edition polls Twitter every hour while the premium edition pulls tweets every 15 minutes so it is more suitable for tracking terms that are generating lot of tweets (like a conference #hashtag or a live sports event).

The one step that I find crucial to ensuring that I have properly archived material to work with is to define the exact date of the Tweets. As you notice, the "Date" column (Column "A") does not have the time.

1) Click on the "A" at the top to highlight the entire column.

2) Go to "Format" > "Number"> select "Date time" from the list.

Your "Date" column is now formatted with the time.

Quick note before we proceed: If you have not installed Google Fusion Tables or you are unfamiliar with how they work, I recommend checking out my Instructables on Create a Clickable Google Map to Share Geographic Information and Metadata: https://www.instructables.com/id/Create-a-Clickable-Google-Map-to-Share-Geographic-/

1) Go back to your drive (https://drive.google.com/). Click on the red button that says "New", go to "More", and click "Google Fusion Tables". Select "Google Spreadsheets"

2) Select "Google Spreadsheets" from the left hand corner

3) Select the Twitter Archiver spreadsheet you have just created and click "Select" (This may take some time to load!)

4) You need to make sure that you are importing the correct sheet from the Twitter Archiver. Make sure that the sheet selected is the one you created (and not the logs) and click "Next>>"

5) There is a box next to some text that says "Column names are in row". You want to make sure that you grab the correct column names, which should be in row 2. Click "Next >>" and then "Finish"

The next step involves telling Fusion Tables where to find the geographic information.

1) Click on the arrow beside your "Location" column, and click on "Change...".
2) Next to "Type" you will see that "Text" may be chosen by default. Click on "Text" and change the type to "Location". Hit "Save".

3) Click on the Red "+" next to "Cards". Select "'Add map".
4) Google Fusion Tables will immediately start geocoding the locations from your spreadsheet, and will produce a map with all the points from your "Location" column. This will take some time. Notice the section that says Quality: You will always get ambiguous results because Twitter allows users to enter whatever they want in the location in their profile.

Congrats on finishing your map. You can click any of the points to pull up the information from the archived tweets!

Now that you have completed your map, check out my other Instructables:

Create a Clickable Google Map to Share Geographic Information and Metadata (Start Here! For Beginners)

Using IFFTTT, Google Sheets, and Google Fusion Tables to create a Live Photo Map

Other useful information:

Want to live sync a Google Sheet to a Google Fusion Table? Check out the documentation here!