EDA | Kishan Mistri

Real-time Trends Insights from Tweet Data

Thu, 06 Oct 2022 08:35:01 +0000

1. Local and global thought patterns

While we might not be Twitter fans, we have to admit that it has a huge influence on the world (who doesn't know about Trump's tweets). Twitter data is not only gold in terms of insights, but Twitter-storms are available for analysis in near real-time. This means we can learn about the big waves of thoughts and moods around the world as they arise.

As any place filled with riches, Twitter has security guards blocking us from laying our hands on the data right away ⛔️ Some authentication steps (really straightforward) are needed to call their APIs for data collection. Since our goal today is learning to extract insights from data, we have already gotten a green-pass from security ✅ Our data is ready for usage in the datasets folder — we can concentrate on the fun part! 🕵️‍♀️🌎

Twitter provides both global and local trends. Let's load and inspect data for topics that were hot worldwide (WW) and in the United States (US) at the moment of query — snapshot of JSON response from the call to Twitter's GET trends/place API.

Note: Here is the documentation for this call, and here a full overview on Twitter's APIs.

# Loading json module
import json

# Load WW_trends and US_trends data into the the given variables respectively
WW_trends = json.loads(open('datasets/WWTrends.json').read())
US_trends = json.loads(open('datasets/USTrends.json').read())

Our data was hard to read! Luckily, we can resort to the json.dumps() method to have it formatted as a pretty JSON string.

# Pretty-printing the results. First WW and then US trends.

print("WW trends:")
json.dumps(WW_trends,indent = 1)

Expand Output

WW trends:
 [
 {
 "trends": [
 {
 "name": "#BeratKandili",
 "url": "http://twitter.com/search?q=%23BeratKandili",
 "promoted_content": null,
 "query": "%23BeratKandili",
 "tweet_volume": 46373
 },
 {
 "name": "#GoodFriday",
 "url": "http://twitter.com/search?q=%23GoodFriday",
 "promoted_content": null,
 "query": "%23GoodFriday",
 "tweet_volume": 81891
 },
 {
 ...
 },
 {
 "name": "#JunquerasACN",
 "url": "http://twitter.com/search?q=%23JunquerasACN",
 "promoted_content": null,
 "query": "%23JunquerasACN",
 "tweet_volume": null
 }
 ],
 "as_of": "2019-04-19T08:43:43Z",
 "created_at": "2019-04-19T08:39:15Z",
 "locations": [
 {
 "name": "Worldwide",
 "woeid": 1
 }
 ]
 }
 ]

print("\n", "US trends:")
json.dumps(US_trends,indent = 1)

Expand Output

US trends:
 [
 {
 "trends": [
 {
 "name": "#WeLoveTheEarth",
 "url": "http://twitter.com/search?q=%23WeLoveTheEarth",
 "promoted_content": null,
 "query": "%23WeLoveTheEarth",
 "tweet_volume": 159698
 },
 {
 "name": "#DragRace",
 "url": "http://twitter.com/search?q=%23DragRace",
 "promoted_content": null,
 "query": "%23DragRace",
 "tweet_volume": 37166
 },
 {
 ...
 },
 {
 "name": "#WeirdDateStories",
 "url": "http://twitter.com/search?q=%23WeirdDateStories",
 "promoted_content": null,
 "query": "%23WeirdDateStories",
 "tweet_volume": null
 },
 {
 "name": "#HustleAndSoul",
 "url": "http://twitter.com/search?q=%23HustleAndSoul",
 "promoted_content": null,
 "query": "%23HustleAndSoul",
 "tweet_volume": null
 },
 {
 "name": "#fridaymotivation",
 "url": "http://twitter.com/search?q=%23fridaymotivation",
 "promoted_content": null,
 "query": "%23fridaymotivation",
 "tweet_volume": null
 }
 ],
 "as_of": "2019-04-19T08:43:43Z",
 "created_at": "2019-04-19T08:39:15Z",
 "locations": [
 {
 "name": "United States",
 "woeid": 23424977
 }
 ]
 }
 ]

3. Finding common trends

🕵️‍♀️ From the pretty-printed results (output of the previous task), we can observe that:

We have an array of trend objects having: the name of the trending topic, the query parameter that can be used to search for the topic on Twitter-Search, the search URL and the volume of tweets for the last 24 hours, if available. (The trends get updated every 5 mins.)
At query time #BeratKandili, #GoodFriday and #WeLoveTheEarth were trending WW.
"tweet_volume" tell us that #WeLoveTheEarth was the most popular among the three.
Results are not sorted by "tweet_volume".
There are some trends which are unique to the US.

It’s easy to skim through the two sets of trends and spot common trends, but let's not do "manual" work. We can use Python’s set data structure to find common trends — we can iterate through the two trends objects, cast the lists of names to sets, and call the intersection method to get the common names between the two sets.

# Extracting all the WW trend names from WW_trends
world_trends = set([WW_trends[0]['trends'][i]['name'] for i in range(len(WW_trends[0]['trends']))])

# Extracting all the US trend names from US_trends
us_trends = set([US_trends[0]['trends'][i]['name'] for i in range(len(US_trends[0]['trends']))])

# Getting the intersection of the two sets of trends
common_trends = world_trends.intersection(us_trends)

# Inspecting the data
world_trends

Expand Output

 {'#195TLdenTTVerilir',
 '#19aprile',
 '#AFLNorthDons',
 '#BLACKPINKxCorden',
 '#BeratKandili',
 '#CHIvLIO',
 '#ConCalmaRemix',
 '#DinahJane1',
 '#DragRace',
 '#DuyguAsena',
 '#GoodFriday',
 '#HanumanJayanti',
 '#HardikPatel',
 '#HayırlıCumalar',
 '#HayırlıKandiller',
 '#IndonesianElectionHeroes',
 '#Jersey',
 '#JunquerasACN',
 '#Karfreitag',
 '#KpuJanganCurang',
 '#NRLBulldogsSouths',
 '#NikahUmurBerapa',
 '#Ontas',
 '#ProtestoEdiyorum',
 '#ShivSena',
 '#TheJudasInMyLife',
 '#ViernesSanto',
 '#WeLoveTheEarth',
 '#اغلاق_BBM',
 '#يوم_الجمعه',
 'Berat Kandilimiz',
 'Derrick White',
 'Derry',
 'Hemant Karkare',
 'Lil Dicky',
 'Lyra McKee',
 'Priyanka Chaturvedi',
 'Shiv Sena',
 'örgütdeğil arkadaşgrubu',
 'グレア',
 'プリウス',
 '免許返納',
 '刀ステ',
 '十二国記',
 '東京・池袋衝突事故',
 '歩行者',
 '池袋の事故',
 '重体の女性と女児',
 '高齢者',
 '브이알'}

us_trends

Expand Output

 {'"Earth"',
 '#AFLNorthDons',
 '#BLACKPINKxCorden',
 '#CUZILOVEYOU',
 '#ConCalmaRemix',
 '#CriticalRoleSpoilers',
 '#DinahJane1',
 '#DontChangeOutNow',
 '#DragRace',
 '#Earth',
 '#FridayFeeling',
 '#GSWvsLAC',
 '#GossipShouldBe',
 '#HustleAndSoul',
 '#LilDicky',
 '#MakeAMovieSensual',
 '#MyDrunkUncleSays',
 '#MyInnerDemonSaid',
 '#NRLBulldogsSouths',
 '#RPDR',
 '#StarTrekDiscovery',
 '#TheLegendOfVoxMachina',
 '#TimeToImpeach',
 '#WGAMIX',
 '#WeLoveTheEarth',
 '#WeirdDateStories',
 '#WhatStopsYouFromGoingHome',
 '#WorldofWarcraftMains',
 '#fridaymotivation',
 '#rupaulsdragrace',
 'David Fletcher',
 'Derrick White',
 'Derry',
 'Gallant',
 'Game 6',
 'George Conway',
 'Kazuo Koike',
 'Kevin Durant',
 'Lil Dicky',
 'Lone Wolf and Cub',
 'Lyra McKee',
 'Mike Anderson',
 'Oshie',
 'Servais',
 'Seth Abramson',
 'Shy Glizzy',
 'Silky',
 'Tomas Hertl',
 'WE LOVE THE EARTH',
 'Yvie'}

print (len(common_trends), "common trends:", common_trends)

Expand Output

\\\\ 11 common trends: {'#ConCalmaRemix', '#BLACKPINKxCorden', 'Derry', 'Lil Dicky', 'Derrick White', '#WeLoveTheEarth', '#DragRace', '#AFLNorthDons', 'Lyra McKee', '#NRLBulldogsSouths', '#DinahJane1'} \\\\\

4. Exploring the hot trend

🕵️‍♀️ From the intersection (last output) we can see that, out of the two sets of trends (each of size 50), we have 11 overlapping topics. In particular, there is one common trend that sounds very interesting: #WeLoveTheEarth — so good to see that Twitteratis are unanimously talking about loving Mother Earth! 💚

Note: We could have had no overlap or a much higher overlap; when we did the query for getting the trends, people in the US could have been on fire obout topics only relevant to them.

Image Source:Official Music Video Cover: https://welovetheearth.org/video/

We have found a hot-trend, #WeLoveTheEarth. Now let's see what story it is screaming to tell us!
If we query Twitter's search API with this hashtag as query parameter, we get back actual tweets related to it. We have the response from the search API stored in the datasets folder as 'WeLoveTheEarth.json'. So let's load this dataset and do a deep dive in this trend.

# Loading the data
tweets = json.loads(open('datasets/WeLoveTheEarth.json').read())

# Inspecting some tweets
tweets[0:2]

Expand Output

 [{'created_at': 'Fri Apr 19 08:46:48 +0000 2019',
 'id': 1119160405270523904,
 'id_str': '1119160405270523904',
 'text': 'RT @lildickytweets: 🌎 out now #WeLoveTheEarth https://t.co/L22XsoT5P1',
 'truncated': False,
 'entities': {'hashtags': [{'text': 'WeLoveTheEarth', 'indices': [30, 45]}],
 'symbols': [],
 'user_mentions': [{'screen_name': 'lildickytweets',
 'name': 'LD',
 'id': 1209516660,
 'id_str': '1209516660',
 'indices': [3, 18]}],
 'urls': [{'url': 'https://t.co/L22XsoT5P1',
 'expanded_url': 'https://youtu.be/pvuN_WvF1to',
 'display_url': 'youtu.be/pvuN_WvF1to',
 'indices': [46, 69]}]},
 'metadata': {'iso_language_code': 'en', 'result_type': 'recent'},
 'source': '<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>',
 'in_reply_to_status_id': None,
 'in_reply_to_status_id_str': None,
 'in_reply_to_user_id': None,
 'in_reply_to_user_id_str': None,
 'in_reply_to_screen_name': None,
 'user': {'id': 212375312,
 'id_str': '212375312',
 'name': 'fake smile',
 'screen_name': 'Pati95Poland',
 'location': 'SWAGLAND ',
 'description': "''you just got knocked the fuck out''",
 'url': 'https://t.co/nxzlSyYZSK',
 'entities': {'url': {'urls': [{'url': 'https://t.co/nxzlSyYZSK',
 'expanded_url': 'http://loveeeujdb.tumblr.com/',
 'display_url': 'loveeeujdb.tumblr.com',
 'indices': [0, 23]}]},
 'description': {'urls': []}},
 'protected': False,
 'followers_count': 2306,
 'friends_count': 697,
 'listed_count': 28,
 'created_at': 'Fri Nov 05 22:25:29 +0000 2010',
 'favourites_count': 5552,
 'utc_offset': None,
 'time_zone': None,
 'geo_enabled': True,
 'verified': False,
 'statuses_count': 185750,
 'lang': 'pl',
 'contributors_enabled': False,
 'is_translator': False,
 'is_translation_enabled': False,
 'profile_background_color': 'FFFFFF',
 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme18/bg.gif',
 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme18/bg.gif',
 'profile_background_tile': False,
 'profile_image_url': 'http://pbs.twimg.com/profile_images/1093929135183937537/hQuxtwKq_normal.jpg',
 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1093929135183937537/hQuxtwKq_normal.jpg',
 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/212375312/1522705183',
 'profile_link_color': 'ABB8C2',
 'profile_sidebar_border_color': 'FFFFFF',
 'profile_sidebar_fill_color': 'F6F6F6',
 'profile_text_color': '333333',
 'profile_use_background_image': True,
 'has_extended_profile': True,
 'default_profile': False,
 'default_profile_image': False,
 'following': False,
 'follow_request_sent': False,
 'notifications': False,
 'translator_type': 'regular'},
 'geo': None,
 'coordinates': None,
 'place': None,
 'contributors': None,
 'retweeted_status': {'created_at': 'Fri Apr 19 04:22:29 +0000 2019',
 'id': 1119093888524754946,
 'id_str': '1119093888524754946',
 'text': '🌎 out now #WeLoveTheEarth https://t.co/L22XsoT5P1',
 'truncated': False,
 'entities': {'hashtags': [{'text': 'WeLoveTheEarth', 'indices': [10, 25]}],
 'symbols': [],
 'user_mentions': [],
 'urls': [{'url': 'https://t.co/L22XsoT5P1',
 'expanded_url': 'https://youtu.be/pvuN_WvF1to',
 'display_url': 'youtu.be/pvuN_WvF1to',
 'indices': [26, 49]}]},
 'metadata': {'iso_language_code': 'en', 'result_type': 'recent'},
 'source': '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>',
 'in_reply_to_status_id': None,
 'in_reply_to_status_id_str': None,
 'in_reply_to_user_id': None,
 'in_reply_to_user_id_str': None,
 'in_reply_to_screen_name': None,
 'user': {'id': 1209516660,
 'id_str': '1209516660',
 'name': 'LD',
 'screen_name': 'lildickytweets',
 'location': 'Earth',
 'description': 'Rapper/Actor/Comedian/Model #WeLoveTheEarth',
 'url': 'https://t.co/aFrPkkJKqs',
 'entities': {'url': {'urls': [{'url': 'https://t.co/aFrPkkJKqs',
 'expanded_url': 'https://LilDicky.lnk.to/Earth',
 'display_url': 'LilDicky.lnk.to/Earth',
 'indices': [0, 23]}]},
 'description': {'urls': []}},
 'protected': False,
 'followers_count': 503111,
 'friends_count': 945,
 'listed_count': 657,
 'created_at': 'Fri Feb 22 19:06:15 +0000 2013',
 'favourites_count': 7696,
 'utc_offset': None,
 'time_zone': None,
 'geo_enabled': False,
 'verified': True,
 'statuses_count': 14430,
 'lang': 'en',
 'contributors_enabled': False,
 'is_translator': False,
 'is_translation_enabled': False,
 'profile_background_color': '000000',
 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme14/bg.gif',
 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme14/bg.gif',
 'profile_background_tile': False,
 'profile_image_url': 'http://pbs.twimg.com/profile_images/1119087366679846912/XSa4fpQA_normal.png',
 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1119087366679846912/XSa4fpQA_normal.png',
 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/1209516660/1555646206',
 'profile_link_color': '858585',
 'profile_sidebar_border_color': 'FFFFFF',
 'profile_sidebar_fill_color': 'DDEEF6',
 'profile_text_color': '333333',
 'profile_use_background_image': True,
 'has_extended_profile': False,
 'default_profile': False,
 'default_profile_image': False,
 'following': False,
 'follow_request_sent': False,
 'notifications': False,
 'translator_type': 'none'},
 'geo': None,
 'coordinates': None,
 'place': None,
 'contributors': None,
 'is_quote_status': False,
 'retweet_count': 7482,
 'favorite_count': 13317,
 'favorited': False,
 'retweeted': False,
 'possibly_sensitive': False,
 'lang': 'en'},
 'is_quote_status': False,
 'retweet_count': 7482,
 'favorite_count': 0,
 'favorited': False,
 'retweeted': False,
 'possibly_sensitive': False,
 'lang': 'en'},
 {'created_at': 'Fri Apr 19 08:46:48 +0000 2019',
 'id': 1119160404876206080,
 'id_str': '1119160404876206080',
 'text': '💚🌎💚 #WeLoveTheEarth 👇🏼',
 'truncated': False,
 'entities': {'hashtags': [{'text': 'WeLoveTheEarth', 'indices': [5, 20]}],
 'symbols': [],
 'user_mentions': [],
 'urls': []},
 'metadata': {'iso_language_code': 'und', 'result_type': 'recent'},
 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
 'in_reply_to_status_id': None,
 'in_reply_to_status_id_str': None,
 'in_reply_to_user_id': None,
 'in_reply_to_user_id_str': None,
 'in_reply_to_screen_name': None,
 'user': {'id': 72150460,
 'id_str': '72150460',
 'name': 'Alfonsina del Mar (Dianishka Prietishka)',
 'screen_name': 'Diana____X',
 'location': 'México',
 'description': '☸︎ ⚓︎ 👩🏻\u200d✈️ ★★★★ ♒︎',
 'url': None,
 'entities': {'description': {'urls': []}},
 'protected': False,
 'followers_count': 223,
 'friends_count': 513,
 'listed_count': 0,
 'created_at': 'Sun Sep 06 23:26:24 +0000 2009',
 'favourites_count': 750,
 'utc_offset': None,
 'time_zone': None,
 'geo_enabled': True,
 'verified': False,
 'statuses_count': 1668,
 'lang': 'es',
 'contributors_enabled': False,
 'is_translator': False,
 'is_translation_enabled': False,
 'profile_background_color': '642D8B',
 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme10/bg.gif',
 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme10/bg.gif',
 'profile_background_tile': True,
 'profile_image_url': 'http://pbs.twimg.com/profile_images/1072296818531278848/tgn0e2h4_normal.jpg',
 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1072296818531278848/tgn0e2h4_normal.jpg',
 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/72150460/1545198367',
 'profile_link_color': '8400FF',
 'profile_sidebar_border_color': '65B0DA',
 'profile_sidebar_fill_color': '7AC3EE',
 'profile_text_color': '3D1957',
 'profile_use_background_image': True,
 'has_extended_profile': True,
 'default_profile': False,
 'default_profile_image': False,
 'following': False,
 'follow_request_sent': False,
 'notifications': False,
 'translator_type': 'none'},
 'geo': None,
 'coordinates': None,
 'place': None,
 'contributors': None,
 'is_quote_status': False,
 'retweet_count': 0,
 'favorite_count': 0,
 'favorited': False,
 'retweeted': False,
 'lang': 'und'}]

5. Digging deeper

🕵️‍♀️ Printing the first two tweet items makes us realize that there’s a lot more to a tweet than what we normally think of as a tweet — there is a lot more than just a short text!

But hey, let's not get overwhemled by all the information in a tweet object! Let's focus on a few interesting fields and see if we can find any hidden insights there.

# Extracting the text of all the tweets from the tweet object
texts = [tweet['text'] for tweet in tweets]

# Extracting screen names of users tweeting about #WeLoveTheEarth
names = [user_mentions['screen_name'] for tweet in tweets for user_mentions in tweet['entities']['user_mentions']]

# Extracting all the hashtags being used when talking about this topic
hashtags = [hashtag['text'] for tweet in tweets for hashtag in tweet['entities']['hashtags']]

# Inspecting the first 10 results
print(json.dumps(texts[0:10], indent=1),"\n")

Expand Output

 [
 "RT @lildickytweets: \ud83c\udf0e out now #WeLoveTheEarth https://t.co/L22XsoT5P1",
 "\ud83d\udc9a\ud83c\udf0e\ud83d\udc9a #WeLoveTheEarth \ud83d\udc47\ud83c\udffc",
 "RT @cabeyoomoon: Ta piosenka to bop, wpada w ucho i dochody z niej id\u0105 na dobry cel, warto s\u0142ucha\u0107 w k\u00f3\u0142ko i w k\u00f3\u0142ko gdziekolwiek si\u0119 ty\u2026",
 "#WeLoveTheEarth \nCzemu ja si\u0119 pop\u0142aka\u0142am",
 "RT @Spotify: This is epic. @lildickytweets got @justinbieber, @arianagrande, @halsey, @sanbenito, @edsheeran, @SnoopDogg, @ShawnMendes, @Kr\u2026",
 "RT @biebercentineo: Justin : are we gonna die? \nLil dicky: you know bieber we might die \n\nBTCH IM CRYING #EARTH #WeLoveTheEarth #WELOVEEART\u2026",
 "RT @dreamsiinflate: #WeLoveTheEarth \u201ci am a fat fucking pig\u201d okay brendon urie https://t.co/FdJmq31xZc",
 "Literally no one:\n\nMe in the past 4 hours:\n\nI'm a koala and I sleep all the time, so what, it's cute \ud83c\udfb6\n\n#WeLoveTheEarth #EdSheeranTheKoala",
 "RT @Yuuupthatsme: Mia\u0142e\u015b by\u0107 \u017cyraf\u0105 #WeLoveTheEarth https://t.co/0kNCpU8o6q",
 "RT @jaguareffects: eu prestando aten\u00e7\u00e3o no \u00e1udio pra identificar cada artista\n\n#WeLoveTheEarth https://t.co/0cDtiV2t1E"
 ]

json.dumps(names[0:10], indent=1)

Expand Output

 '[\n "lildickytweets",\n "cabeyoomoon",\n "Spotify",\n "lildickytweets",\n "justinbieber",\n "ArianaGrande",\n "halsey",\n "sanbenito",\n "edsheeran",\n "SnoopDogg"\n]'

json.dumps(hashtags[0:10], indent=1)

Expand Output

 '[\n "WeLoveTheEarth",\n "WeLoveTheEarth",\n "WeLoveTheEarth",\n "EARTH",\n "WeLoveTheEarth",\n "WeLoveTheEarth",\n "WeLoveTheEarth",\n "EdSheeranTheKoala",\n "WeLoveTheEarth",\n "WeLoveTheEarth"\n]'

6. Frequency analysis

🕵️‍♀️ Just from the first few results of the last extraction, we can deduce that:

We are talking about a song about loving the Earth.
A lot of big artists are the forces behind this Twitter wave, especially Lil Dicky.
Ed Sheeran was some cute koala in the song — "EdSheeranTheKoala" hashtag! 🐨

Observing the first 10 items of the interesting fields gave us a sense of the data. We can now take a closer look by doing a simple, but very useful, exercise — computing frequency distributions. Starting simple with frequencies is generally a good approach; it helps in getting ideas about how to proceed further.

# Importing modules
from collections import Counter

# Counting occcurrences/ getting frequency dist of all names and hashtags
for item in [names, hashtags]:
 c = Counter(item)
 # Inspecting the 10 most common items in c
 print (c.most_common(10), "\n")

Expand Output

 [('lildickytweets', 102), ('LeoDiCaprio', 44), ('ShawnMendes', 33), ('halsey', 31), ('ArianaGrande', 30), ('justinbieber', 29), ('Spotify', 26), ('edsheeran', 26), ('sanbenito', 25), ('SnoopDogg', 25)]

 [('WeLoveTheEarth', 313), ('4future', 12), ('19aprile', 12), ('EARTH', 11), ('fridaysforfuture', 10), ('EarthMusicVideo', 3), ('ConCalmaRemix', 3), ('Earth', 3), ('aliens', 2), ('AvengersEndgame', 2)]

7. Activity around the trend

🕵️‍♀️ Based on the last frequency distributions we can further build-up on our deductions:

We can more safely say that this was a music video about Earth (hashtag 'EarthMusicVideo') by Lil Dicky.
DiCaprio is not a music artist, but he was involved as well (Leo is an environmentalist so not a surprise to see his name pop up here).
We can also say that the video was released on a Friday; very likely on April 19th.

We have been able to extract so many insights. Quite powerful, isn't it?!

Let's further analyze the data to find patterns in the activity around the tweets — did all retweets occur around a particular tweet?

If a tweet has been retweeted, the 'retweeted_status' field gives many interesting details about the original tweet itself and its author.

We can measure a tweet's popularity by analyzing the retweetcount and favoritecount fields. But let's also extract the number of followers of the tweeter — we have a lot of celebs in the picture, so can we tell if their advocating for #WeLoveTheEarth influenced a significant proportion of their followers?

Note: The retweet_count gives us the total number of times the original tweet was retweeted. It should be the same in both the original tweet and all the next retweets. Tinkering around with some sample tweets and the official documentaiton are the way to get your head around the mnay fields.

# Extracting useful information from retweets
# tweets[0]['text']
retweets = [(tweet['retweet_count'], tweet['retweeted_status']['favorite_count'], tweet['retweeted_status']['user']['followers_count'], tweet['retweeted_status']['user']['screen_name'], tweet['text']) for tweet in tweets if 'retweeted_status' in tweet]
retweets

Expand Output

 [(7482,
 13317,
 503111,
 'lildickytweets',
 'RT @lildickytweets: 🌎 out now #WeLoveTheEarth https://t.co/L22XsoT5P1'),
 (9,
 23,
 7732,
 'cabeyoomoon',
 'RT @cabeyoomoon: Ta piosenka to bop, wpada w ucho i dochody z niej idą na dobry cel, warto słuchać w kółko i w kółko gdziekolwiek się ty…'),
 (4288,
 9488,
 2973277,
 'Spotify',
 'RT @Spotify: This is epic. @lildickytweets got @justinbieber, @arianagrande, @halsey, @sanbenito, @edsheeran, @SnoopDogg, @ShawnMendes, @Kr…'),
 (517,
 1185,
 4038,
 'biebercentineo',
 'RT @biebercentineo: Justin : are we gonna die? \nLil dicky: you know bieber we might die \n\nBTCH IM CRYING #EARTH #WeLoveTheEarth #WELOVEEART…'),
 (369,
 1142,
 3045,
 'dreamsiinflate',
 'RT @dreamsiinflate: #WeLoveTheEarth “i am a fat fucking pig” okay brendon urie https://t.co/FdJmq31xZc'),
 (8,
 56,
 669,
 'Yuuupthatsme',
 'RT @Yuuupthatsme: Miałeś być żyrafą #WeLoveTheEarth https://t.co/0kNCpU8o6q'),
 (
 ...
 ),
 (493,
 2095,
 39726,
 'MendesNotified',
 'RT @MendesNotified: shawn reading his line for the song #WeLoveTheEarth: https://t.co/dcWJFDlVMd'),
 (3,
 6,
 9693,
 'TJOfficial_',
 'RT @TJOfficial_: We love the earth it is our planet 🌍 🎶 #WeLoveTheEarth It’s so beautiful.'),
 (89,
 121,
 2124,
 'kingoftheclout',
 'RT @kingoftheclout: STOP SCROLLING AND WATCH THIS VIDEO‼️ please take some time out of your day to watch this video &amp; possibly share it. it…')]

8. A table that speaks a 1000 words

Let's manipulate the data further and visualize it in a better and richer way — "looks matter!"

# Importing modules
import matplotlib.pyplot as plt
import pandas as pd

# Create a DataFrame and visualize the data in a pretty and insightful format
df = pd.DataFrame(retweets, columns=['Retweets','Favorites', 'Followers', 'ScreenName', 'Text'])

df = df.groupby(['ScreenName','Text','Followers']).sum().sort_values(by = ['Followers'],ascending=[False])
df.style.background_gradient()

Expand Output

Tweets Object	Retweets	Favorites
(‘katyperry’, ‘RT @katyperry: Sure, the Mueller report is out, but @lildickytweets’ “Earth” but will be too tonight. Don't say I never tried to save the w…’, 107195569)	2338	10557
(‘katyperry’, ‘RT @katyperry: Sure, the Mueller report is out, but @lildickytweets’ “Earth” but will be too tonight. Don't say I never tried to save the w…’, 107195568)	2338	10556
(‘TheEllenShow’, ‘RT @TheEllenShow: .@lildickytweets, @justinbieber, @MileyCyrus, @katyperry, @ArianaGrande and more, all in one music video. My head might e…’, 77474826)	2432	10086
(‘LeoDiCaprio’, ‘RT @LeoDiCaprio: The @dicapriofdn partners that will benefit from this collaboration include @SharkRayFund, @SolutionsProj, @GreengrantsFun…’, 18988898)	28505	78137
(‘LeoDiCaprio’, ‘RT @LeoDiCaprio: Thank you to @lildickytweets and all the artists that came together to make this happen. Net profits from the song, video,…’, 18988898)	149992	416018
(‘halsey’, ‘RT @halsey: 🐯🐯🐯 meeeeeeeow 👅 #WeLoveTheEarth https://t.co/JJr6vmKG7U', 10564842)	7742	69684
(‘halsey’, ‘RT @halsey: 🐯🐯🐯 meeeeeeeow 👅 #WeLoveTheEarth https://t.co/JJr6vmKG7U', 10564841)	7742	69682
(‘scooterbraun’, ‘RT @scooterbraun: Watch #EARTH NOW! #WeLoveTheEarth — https://t.co/OtRfIgcSZL', 3885179)	5925	14635
(‘Spotify’, ‘RT @Spotify: This is epic. @lildickytweets got @justinbieber, @arianagrande, @halsey, @sanbenito, @edsheeran, @SnoopDogg, @ShawnMendes, @Kr…’, 2973277)	107222	237259
(‘TomHall’, ‘RT @TomHall: 🐆\n\nLook Out!\n\n🐆\n\n#Cheetah @LeoDiCaprio #FridayFeeling #WeLoveTheEarth \n\nhttps://t.co/w1O1NyTbwX’, 590841)	82	252
(’lildickytweets’, ‘RT @lildickytweets: Earth. 4/18 at 9PM PST. #WeLoveTheEarth pre-save link in my bio https://t.co/7HD2xlSFYQ', 503112)	25098	90974
(’lildickytweets’, ‘RT @lildickytweets: 🌎 out now #WeLoveTheEarth https://t.co/L22XsoT5P1', 503112)	112230	199773
(’lildickytweets’, ‘RT @lildickytweets: 🌎 out now #WeLoveTheEarth https://t.co/L22XsoT5P1', 503111)	119712	213072
(‘greenpeacefr’, ‘RT @greenpeacefr: Happening NOW in France : today, to show how much #WeLoveTheEarth, more than 2000 citizens are blocking the headquarters…’, 420789)	128	192
(‘rlthingy’, ‘RT @rlthingy: /rlt/ YO GUYS LISTEN TO THIS SONGGGGG #WeLoveTheEarth https://t.co/trUgRG7QBH', 225977)	11	23
(‘SB_Projects’, ‘RT @SB_Projects: #WeLoveTheEarth out now with @lildickytweets, @justinbieber, @ArianaGrande, @zacbrownband, and 25+ of your other faves htt…’, 98418)	271	730
(‘biebernovidade’, ‘RT @biebernovidade: Rio de Janeiro: PRESENTE! #WeLoveTheEarth https://t.co/IJrypKqzSf', 83281)	651	1499
(‘MendesCrewInfo’, ‘RT @MendesCrewInfo: .@ShawnMendes represents rhinos in the ‘Earth’ music video and feature. His line, “We’re just some rhinos, horny as hec…’, 61015)	2815	9599
(‘shawnxniallx’, ‘RT @shawnxniallx: La gente piensa que el calentamiento global no es real, ya abran los ojos y actuemos de una manera real, dime ¿qué te cue…’, 53713)	812	1254
(‘shawnxniallx’, ‘RT @shawnxniallx: La humanidad es un asco y nunca me cansaré de decirlos ¿no creen que es momento de cambiar? Salvemos nuestro planeta, sal…’, 53713)	604	1034
(‘shawnxniallx’, ‘RT @shawnxniallx: Vengo a recomendarles a todos esta miniserie que realizó Netflix, en la cual nos muestra cada parte de las especies, lo q…’, 53713)	138	299
(‘shawnxniallx’, ‘RT @shawnxniallx: Espero que con este video se pueda crear un poco más de conciencia propia, simeplemnte estamos acabando con nuestro plane…’, 53713)	300	494
(‘SMendesQandA’, ‘RT @SMendesQandA: #WeLoveTheEarth https://t.co/KDIKYVHFWU', 53282)	56	339
(‘SMendesQandA’, ‘RT @SMendesQandA: Let’s help save the Earth. We’re the ones with the power of change. 🌍 #WeLoveTheEarth’, 53282)	122	343
(‘SMendesQandA’, ‘RT @SMendesQandA: So you can basically say you have a song with many of the artists we have asked you to collaborate with…smart move mend…’, 53282)	116	717
(‘SMendesQandA’, ‘RT @SMendesQandA: “We’re just some rhinos, horny as heck” — Shawn’s part on #WeLoveTheEarth’, 53282)	244	1420
(‘SMendesQandA’, ‘RT @SMendesQandA: “We’re just some rhinos horny as heck” — Shawn Mendes #WeLoveTheEarth https://t.co/URnHb0DTWN', 53282)	768	3309
(‘ShawnUpdatesSA’, “RT @ShawnUpdatesSA: Shawn Mendes: rinocerontes.\n#WeLoveTheEarth\n\n’La población de rinocerontes negros está casi extinta, y las otras cuatro…”, 52424)	850	1807
(‘gtbriel’, ‘RT @gtbriel: vimos o cu do justin bieber\n\nlogo em seguida a ariana sair do cu do justin \n\ne logo em seguuda a ariana sendo morta pela halse…’, 49215)	581	970
(‘MileySmilerNews’, ‘RT @MileySmilerNews: Miley’s cameo in the Earth song is so cute #WeLoveTheEarth https://t.co/Byk9rkhMpG', 47840)	15	116
(‘ArianatorFallen’, ‘RT @ArianatorFallen: Ariana vocals had saved the earth . This song is so important #WeLoveTheEarth https://t.co/Lg9jsPRD5z', 44426)	84	258
(‘MendesNotified’, ‘RT @MendesNotified: Rhinos - Shawn Mendes🦏🖤\n“The black rhino population is nearly extinct, and the four other species of rhinos found in Af…’, 39726)	258	858
(‘MendesNotified’, ‘RT @MendesNotified: shawn reading his line for the song #WeLoveTheEarth: https://t.co/dcWJFDlVMd', 39726)	2465	10475
(‘badweputation’, ‘RT @badweputation: n adianta vcs comentarem sobre a importância da preservação hj e amanhã já esquecerem e tbm lembrando q o atual presiden…’, 38787)	119	183
(‘isbreakls’, ‘RT @isbreakls: Queria dizer que na música do Lil Dicky tem 28 artistas incríveis mais quem eu amo mesmo é um babuíno chamado Justin Bieber.…’, 37473)	95	215
(‘isbreakls’, ‘RT @isbreakls: A música e o clipe tá incrível vamos espalhar o verdadeiro motivo da música que é fazer com que as pessoas pensem nos seus a…’, 37473)	684	1068
(‘NoticiasSmilers’, ‘RT @NoticiasSmilers: Soy un elefante, tengo basura en mi trompa. -Pequeño cameo de Miley en la canción Earth de Lil Dicky- #WeLoveTheEarth…’, 32649)	7	4
(‘ThrowbacksBTS’, ‘RT @ThrowbacksBTS: Friendly reminder to use less plastic, recycle and contribute in some way to benefit humanity. The world is dying we nee…’, 32498)	370	816
(‘btsargento’, ‘RT @btsargento: ▪ estas son varias cosas que podemos hacer para construir un planeta mejor, sin contaminación y un lugar donde todos aporte…’, 31453)	67	143
(‘biebersmaniabrs’, ‘RT @biebersmaniabrs: SAIU! Ouçam e assistam “Earth”, música de Lil Dicky com Justin Bieber e mais 29 artistas. 🌏🎶\n\nVideo: https://t.co/oHRR…’, 26097)	341	463
(’landsrauhl’, ‘RT @landsrauhl: I loved what lil dicky did. There’s so many popular artists on the track which hopefully means more of a variety of people…’, 25071)	128	306
(‘biebsrell’, ‘RT @biebsrell: J: ¿Vamos a morir?\nL: ¿Sabes qué, Bieber? Podríamos morir. No voy a mentirte. Quiero decir, hay mucha gente ahí afuera que n…’, 23301)	730	1344
(‘ShawnNewsPoland’, ‘RT @ShawnNewsPoland: “We’re just some rhinos, horny as heck” — Shawna tekst w piosence #WeLoveTheEarth https://t.co/NsOmYfoe7u', 22067)	97	523
(‘ShawnNewsPoland’, ‘RT @ShawnNewsPoland: Wyjaśnienie tekstu Shawna #WeLoveTheEarth https://t.co/NiShcC8WGW', 22067)	105	440
(‘ShawnNewsPoland’, ‘RT @ShawnNewsPoland: Słuchajcie wszędzie gdzie się da! #WeLoveTheEarth dochód z piosenki zostanie przekazany fundacji założonej przez DiCap…’, 22067)	4410	8720
(‘NLiddle16’, ‘RT @NLiddle16: #Earth isn’t supposed to break records or chart. This song is to bring awareness. We have to change now. We have to love mor…’, 21180)	27	134
(‘ArianaRenewsFR’, ‘RT @ArianaRenewsFR: #WeLoveTheEarth est maintenant disponible sur toutes les plateformes ! \n\nYouTube : https://t.co/zSpCdE74lV\n\niTunes : ht…’, 20794)	5	20
(‘iBeliebersMx’, ‘RT @iBeliebersMx: Justin: ¿Vamos a morir?\n\nLil Dicky: “Sabes Bieber… podemos morir. No te voy a mentir a ti, hay muchas personas que pien…’, 19745)	2236	3872
(‘JBCrewdotcom’, ‘RT @JBCrewdotcom: #WeLoveTheEarth Merchandise available to purchase with proceeds going to The Leonardo DiCaprio Foundation dedicated to th…’, 17812)	89	186
(‘dimplecabello’, ‘RT @dimplecabello: “¿Que vamos a defender? El amor. Y nosotros amamos la tierra.” Este video es precioso, últimamente estamos dañando tanto…’, 17590)	286	445
(‘TeamAGPoland’, ‘RT @TeamAGPoland: Utwór “Earth” jest już dostępny! Ariana wcieliła się w postać zebry, a cały dochód ze sprzedaży utworu zostanie przekazan…’, 17157)	44	142
(‘bieberpakiss’, ‘RT @bieberpakiss: HI IM A BABOON IM LIKE A MAN JUST LESS ADVANCED AND MY ANUS IS HUGE #WeLoveTheEarth https://t.co/tqhfvv7PKf', 16288)	148	286
(‘yourbiebernews’, ‘RT @yourbiebernews: Justin Bieber: Baboon\n\n#WeLoveTheEarth https://t.co/dlvl1M28Mg', 14121)	198	404
(‘bizzleslovej’, ‘RT @bizzleslovej: Parliamo di Leo DiCaprio che entra in scena sulla nave del Titanic stile Jack #WeLoveTheEarth https://t.co/VxZscAiOhZ', 13151)	5	8
(‘holdtightlive’, ‘RT @holdtightlive: THIS SONG IS SO BEAUTIFUL THE VIDEO IS ABSOLUTELY AMAZING AND IT’S FOR A GOOD CAUSE SO LETS GOOO STREAM IT BUY IT WATCH…’, 10331)	35	83
(’needyellie’, ‘RT @needyellie: please keep steaming this song, all proceeds go to an extremely important cause. save planet earth!!\n#WeLoveTheEarth https:…’, 10319)	1240	2884
(‘gladwithmyidols’, ‘RT @gladwithmyidols: So… Halsey killed Ariana #WeLoveTheEarth https://t.co/Qquu4YfgzE', 9987)	23	91
(‘TJOfficial’, ‘RT @TJOfficial: We love the earth it is our planet 🌍 🎶 #WeLoveTheEarth It’s so beautiful.’, 9693)	6	12
(‘MCyrusforever’, ‘RT @MCyrusforever: Cały dochód z tej piosenki trafi do fundacji Leonardo DiCaprio, poświęconej ochronie i dobrobytowi wszystkich mieszkań…’, 9680)	46	79
(‘MCyrusforever’, ‘RT @MCyrusforever: Ten teledysk jest świetny, a piosenka ma naprawdę bardzo ważne przesłanie 🌎🌱 #WeLoveTheEarth’, 9680)	1	5
(‘justinxrespect’, ‘RT @justinxrespect: “are we gonna die” ay loco. #WeLoveTheEarth https://t.co/zwof29P4Uw', 9257)	9	21
(‘cyruseyes’, ‘RT @cyruseyes: Qui non stiamo parlando abbastanza di Leonardo DiCaprio e di questa scena \n#WeLoveTheEarth https://t.co/zUfOWBy7ct', 9148)	22	148
(‘cyruseyes’, ‘RT @cyruseyes: Questo video deve diventare un film. Il messaggio arriverebbe veramente a tutti. @ Disney sai cosa fare \n#WeLoveTheEarth ht…’, 9148)	10	52
(‘ChartBTS’, ‘RT @ChartBTS: “We love the earth it is our planet \nWe love the earth it is our home” \n\nARMYs, support this important cause, it is our home…’, 8527)	705	1731
(‘MileyDimension’, ‘RT @MileyDimension: I think everyone should see this! \nIt is about time to do something. #WeLoveTheEarth \nhttps://t.co/WmMDhsBq19’, 7745)	645	969
(‘cabeyoomoon’, ‘RT @cabeyoomoon: Ta piosenka to bop, wpada w ucho i dochody z niej idą na dobry cel, warto słuchać w kółko i w kółko gdziekolwiek się ty…’, 7732)	9	23
(‘stonyproud’, ‘RT @stonyproud: agora que percebi que o clipe todo foi no cu do Justin pqp que cu arrombado #WeLoveTheEarth https://t.co/I2edu6J9be', 7728)	49	115
(‘hoe4exhoee’, “RT @hoe4exhoee: Lil Dicky - Earth (Official Music Video) https://t.co/XT4xdm9XoN \n\nKRIS WU’s PART IS ON 5:43. It is short but I’m so glad…”, 6870)	10	8
(‘KCUnidosBR’, ‘RT @KCUnidosBR: Quem diria em KatyCats? 2 músicas no mesmo dia #ConCalmaRemix #WeLoveTheEarth https://t.co/FF7Be15lhz', 6458)	59	96
(‘Ari_grandeJP’, ‘RT @Ari_grandeJP: . @lildickytweets の\n新曲 #EARTH が公開されました💙🌏\n\nアリアナはシマウマとして参加してます！👀💗✨\n\n #WeLoveTheEarth https://t.co/5g1clvAZoO', 6141)	44	306
(‘becauseihadyou’, ‘RT @becauseihadyou: Don’t know why people are being nasty about ‘Earth’. It’s not a song meant to chart or break records, it’s to raise awa…’, 5647)	2484	6444
(‘biebersmybro’, ‘RT @biebersmybro: beliebers are promoting the song the hardest lol wbk we are the best fandom ever #WeLoveTheEarth’, 5314)	14	49
(‘biebersmybro’, ‘RT @biebersmybro: WE FINALLY HEARD JUSTIN AND ARIANA IN ONE SONG #WeLoveTheEarth https://t.co/RqHTRButmx', 5314)	350	996
(‘imagiwation’, ‘RT @imagiwation: entre tanta coisa ruim e tóxica que tá tendo ultimamente a gente tem a chance de se unir por uma causa boa, então não vamo…’, 5208)	88	129
(’triedsadlonel’, ‘RT @triedsadlonel: piosenka mega mi sie podoba,taka fajna wesoła(pod katem melodii) natomiast calosc mnie hitnela mocno,bo to serio jest pr…’, 4941)	10	34
(’triedsadlonel’, ‘RT @triedsadlonel: kiedy kazdy byl przekonany ze Shawn bedzie żyrafa a tu jeb nosorożcem #WeLoveTheEarth’, 4941)	1	10
(’neznayuchtotut’, ‘RT @neznayuchtotut: вау, так много артистов объединились, чтобы поднять насущную проблему нашей планеты о том, что ее нужно защищать и люби…’, 4621)	8	150
(‘demisfakesmile’, ‘RT @demisfakesmile: stream. stream. stream this song. its for a good cause, let’s save the earth 🌍 💙💚 #WeLoveTheEarth https://t.co/vgABUfce…’, 4421)	504	984
(‘SoundOfSeries’, “RT @SoundOfSeries: #INFOSOS : \n’WeLoveTheEarth’ est disponible sur YouTube. Le but de cette chanson n’est pas de devenir le tube de l’été,…”, 4393)	80	72
(‘carawcciolo’, ‘RT @carawcciolo: Halsey zjada Ariane na śniadanie (koloryzowane) 😅 #WeLoveTheEarth https://t.co/FV4MZSOXlu', 4246)	63	194
(‘gcldendays’, ‘RT @gcldendays: Be careful who you call ugly in middle school 😉\n#WeLoveTheEarth https://t.co/4jvr8Xn397', 4122)	394	1698
(‘gcldendays’, ‘RT @gcldendays: No one: \nNot even a soul:\n\nMe:\n#WeLoveTheEarth https://t.co/i4BxsXBwny', 4122)	408	1189
(‘biebercentineo’, ‘RT @biebercentineo: Justin : are we gonna die? \nLil dicky: you know bieber we might die \n\nBTCH IM CRYING #EARTH #WeLoveTheEarth #WELOVEEART…’, 4038)	1551	3555
(‘biebercentineo’, ‘RT @biebercentineo: THIS SONG IS SO GOOD 😭 #EARTH #WeLoveTheEarth https://t.co/3hM7ZU2eYh', 4038)	79	215
(‘Bizzbiebsz’, ‘RT @Bizzbiebsz: So after so many years we finally got Justin x Arina collab and they even gave us zebra and baboon\n\nStream #WeloveTheEarth…’, 3889)	297	879
(‘jvkejams’, ‘RT @jvkejams: I feel like the song will showcase the animals in a very comedic way (bc its how most lil dicky’s songs are) to raise awarene…’, 3861)	20	104
(’larryxcolours’, ‘RT @larryxcolours: ¿Cómo pueden ayudar a la organización? \n-Compren la mercancia, las ganancias irán directamente para ayudar a la tierra.…’, 3766)	330	612
(‘sitevolts’, ‘RT @sitevolts: CROSSOVER MAIOR QUE VINGADORES 😵 Justin Bieber, Ariana Grande, Katy Perry, Halsey, Sia, Ed Sheeran, Meghan Trainor e Miley C…’, 3707)	130	366
(‘JDBieberPol’, ‘RT @JDBieberPol: Gwiazdy jako zwierzęta w teledysku #Earth #WeLoveTheEarth https://t.co/3aA4mE56Y3', 3475)	34	99
(’nervous_taste’, ‘RT @nervous_taste: Have u thought that we will have:\n\nSHAWN MENDES FT ARIANA GRANDE \nSHAWN MENDES FT JUSTIN BIEBER \nSHAWN MENDES FT HALSEY…’, 3283)	2	9
(’nervous_taste’, ‘RT @nervous_taste: Shawn isn’t the giraffe \n\n#WeLoveTheEarth https://t.co/0PJdZnkP87', 3283)	2	0
(’nopromisesv’, ‘RT @nopromisesv: A ogólnie sam teledysk jest świetny, nie spodziewałam się tego #WeLoveTheEarth’, 3283)	7	77
(‘INLOVEWITHDS’, ‘RT @INLOVEWITHDS: Questo video è semplicemente spettacolare e il messaggio che vuole trasmettere è bellissimo\nwow non ho parole\nTutti gli a…’, 3153)	14	49
(‘agbkissy’, ‘RT @agbkissy: all of yall saying earth is a bad song are immature… its not supposed to be the next big hit, it was made to spread awarene…’, 3098)	584	1356
(‘dreamsiinflate’, ‘RT @dreamsiinflate: #WeLoveTheEarth “i am a fat fucking pig” okay brendon urie https://t.co/FdJmq31xZc', 3045)	1107	3426
(‘dreamsiinflate’, ‘RT @dreamsiinflate: #WeLoveTheEarth they are all these informative cards on the website which are pretty cool https://t.co/qq0GWJ0q0G', 3045)	208	840
(‘SMendesMedia’, ‘RT @SMendesMedia: . @ShawnMendes’ part on l “Earth” 🌎 \n\n#WeLoveTheEarth (@lildickytweets)\n• https://t.co/XrpJbZ6zY7 https://t.co/o2dO7V5cz4', 2815)	238	1120
(‘kyzztin’, ‘RT @kyzztin: The whole concept of this is super dope. lil dicky really got so many huge artists to be part of this amazing project and give…’, 2641)	21	79
(‘kyzztin’, ‘RT @kyzztin: same energy #WeloveTheEarth https://t.co/YfTTS6BRiC', 2641)	10	27
(‘xKittyMendes’, ‘RT @xKittyMendes: SHAWN JEST NOSOROŻCEM\nJego od razu poznałam, bo z innymi miałam problem XD #WeLoveTheEarth https://t.co/21ICMTDdmn', 2597)	48	426
(‘xKittyMendes’, ‘RT @xKittyMendes: Nie sądziłam, że potrzebuję usłyszeć z ust Shawna słowa horny XDD #WeLoveTheEarth https://t.co/TkMA07iNj5', 2597)	326	1074
(‘kajagafelgaja’, ‘RT @kajagafelgaja: macie tu moją przemowe #WeLoveTheEarth https://t.co/1KKrPIRllX', 2526)	1	0
(‘kajagafelgaja’, ‘RT @kajagafelgaja: czy to Was nie dociera że ta piosenka nie ma być dobra, tylko prosta, trafiająca do wszystkich i wpadająca w ucho? i moż…’, 2526)	30	32
(’luvmyburito’, “RT @luvmyburito: głosy ariany i justina tak do siebie pasują ze i’m in shook\n#WeLoveTheEarth https://t.co/MLmyex9t1t", 2520)	90	405
(’lordefurler’, ‘RT @lordefurler: Sia did that! #WeLoveTheEarth https://t.co/aETvluquXM', 2488)	4	26
(‘sinsnvices’, ‘RT @sinsnvices: utożsamiam się z brendonem #WeLoveTheEarth https://t.co/bSLoyWD4hd', 2478)	1646	3466
(‘kingoftheclout’, ‘RT @kingoftheclout: STOP SCROLLING AND WATCH THIS VIDEO‼️ please take some time out of your day to watch this video & possibly share it. it…’, 2124)	178	242
(‘izrnsrdn’, ‘RT @izrnsrdn: Justin Drew Bieber as a baboon \n#WeLoveTheEarth https://t.co/Wz8YK6r21q', 2034)	223	615
(‘marquezduo’, ‘RT @marquezduo: jakby ktoś się zastanawiał jak wygląda moje życie #WeLoveTheEarth https://t.co/SLvLE6eY0V', 1906)	582	1389
(‘5SecOfMendess’, ‘RT @5SecOfMendess: Ejjj ale mi się ta piosenka serio podoba i ma genialne przesłanie!💕\n#WeLoveTheEarth’, 1870)	8	35
(‘JBwakeup’, ‘RT @JBwakeup: Ogolnie to serio glos Justina i Ariany pasuja do siebie.. chce ich colab #WeLoveTheEarth’, 1842)	14	58
(‘MNotifiedMedia’, ‘RT @MNotifiedMedia: Shawn Mendes the Rhino🦏 #WeLoveTheEarth https://t.co/Y2plDHsIsL', 1805)	93	416
(‘MNotifiedMedia’, ‘RT @MNotifiedMedia: “we’re just some rhinos horny as heck” THE RHINOS ARE SO CUTE🦏 #WeLoveTheEarth https://t.co/vvkaO2U7qj', 1805)	175	647
(‘DOLANVVY’, ‘RT @DOLANVVY: Rozjebał mnie snoop dog jako marihuana, rola idealna dla niego\n #WeLoveTheEarth’, 1803)	44	140
(‘justinsbicep’, “RT @justinsbicep: I signed the petition. And you should too. Stream, donate and spread the message. Let’s save the earth. #WeLoveTheEarth h…”, 1772)	58	101
(‘purpxs3bizzle’, ‘RT @purpxs3bizzle: “O homem destrói a natureza com desculpa de sobreviver, a natureza luta para sobreviver para garantir a sobrevivência do…’, 1716)	144	224
(‘kba_yankee21’, ‘RT @kba_yankee21: Never thought I would tear up because of a Lil Dicky song. But here I am crying in a cool way. Feel free to visit https:/…’, 1674)	4	6
(‘beth_powell1’, ‘RT @beth_powell1: LETS SAVE THE PLANET #WeLoveTheEarth’, 1629)	1	0
(‘piggzyneedy’, ‘RT @piggzyneedy: i see so many people tweeting about saving the planet but i bet my whole life the majority of those people are only tweeti…’, 1577)	1	2
(‘particularangel’, ‘RT @particularangel: i mean we should all listen to this part specifically #WeLoveTheEarth https://t.co/tgGZKRgXV2', 1511)	1147	2259
(‘avisbelieberboy’, ‘RT @avisbelieberboy: De lo mejor que va del año #WeLoveTheEarth https://t.co/vN7wPOGork', 1367)	3	4
(‘biebrforro’, ‘RT @biebrforro: justin y ariana cantando juntos es lo mejor que puede existir #WeLoveTheEarth https://t.co/Dilro999ds', 1352)	292	626
(‘shawnnsbabyy’, ‘RT @shawnnsbabyy: Il rinoceronte con la voce più bella del mondo🥰\n\n#WeLoveTheEarth https://t.co/aSQgvumgKl', 1314)	2	6
(‘br_atd’, ‘RT @br_atd: O melhor porco gordo do universo !!!\n\n #WeLoveTheEarth https://t.co/lJfZkgnkWI', 1249)	1	1
(‘moarjustln’, “RT @moarjustln: can’t believe we listened to justin’s voice after months and I still in love \n#WeLoveTheEarth https://t.co/JhFRENhOH3", 1216)	58	158
(‘AGPLOfficial’, ‘RT @AGPLOfficial: Nawet gdy jest zebrą, Ariana zawsze serwuje wokal! 🎶 #WeLoveTheEarth \n\nPosłuchaj utworu “Earth” z Arianą Grande i plejadą…’, 1152)	52	166
(‘xmendesly’, ‘RT @xmendesly: stream earth by lil dicky\n#WeLoveTheEarth’, 1129)	1	0
(‘sourdieselzain’, ‘RT @sourdieselzain: “I’m a fat fucking pig” no ja się tu poplacze zaraz \n#WeLoveTheEarth #EarthMusicVideo https://t.co/tpBQZeRQEh', 1048)	34	99
(‘MrtummyV’, ‘RT @MrtummyV: This is so cute!! aslskfjdkskskkskak\n#WeLoveTheEarth\n\n https://t.co/ZKKKNFQ1Wf', 1005)	10	45
(‘ringsofmendes’, ‘RT @ringsofmendes: głos ari tutaj >>>>>>>>>\n#WeLoveTheEarth https://t.co/QDBVWZ0aom', 1005)	22	119
(‘chimmybby’, ‘RT @chimmybby: We Also Love The Sun 🌞\n#WeLoveTheEarth https://t.co/mh9gfO6Z30', 1004)	34	129
(‘PManice’, ‘RT @PManice: “I’m a koala and I sleep all the time. So what? It’s cute.” 😍 #WeLoveTheEarth https://t.co/vyOvLY4bSE', 979)	33	107
(‘awanasghena’, ‘RT @awanasghena: KATY IN TENDENZA IN ITALIA. PIANGO😭🔥❤️\n#ConCalmaRemix #WeLoveTheEarth https://t.co/CGnsmSd1gM', 895)	7	14
(‘opssmymendes’, ‘RT @opssmymendes: bardzo mi się podoba to, że można kupić merch i pieniądze z tego idą na ratowanie naszej planety #WeLoveTheEarth’, 849)	1	2
(‘opssmymendes’, ‘RT @opssmymendes: warto sobie wejść na stronę https://t.co/JyL7e4jrbv i poczytać o co chodzi z każdym zwierzęciem #WeLoveTheEarth’, 849)	84	112
(’Sunfflower’, ‘RT @Sunfflower: TA PIOSENKA TO ZŁOTO I DIAMENTY\nOby jej przekaz nie skończył się tylko tylko na tym, że ludzie będą ją kojarzyli, bo spok…’, 784)	164	480
(‘jaguareffects’, ‘RT @jaguareffects: eu prestando atenção no áudio pra identificar cada artista\n\n#WeLoveTheEarth https://t.co/0cDtiV2t1E', 771)	253	380
(‘cerberusgrande’, ‘RT @cerberusgrande: wszyscy mówią jak bardzo chcieliby usłyszeć biebera i ariane (same), ale ja bym chciała usłyszeć ariana x halsey, którą…’, 759)	3	7
(‘suplisamanobal’, ‘RT @suplisamanobal: Not blackpink related but Lil dicky x justin bieber, ft ariana grande, and shawn mendes, also charlie puth, and sia and…’, 740)	2	6
(‘Yuuupthatsme’, ‘RT @Yuuupthatsme: Jestem pod wrażeniem, że lil dicky był w stanie pokazać tak poważny problem w taki przyjemny sposób\n#WeLoveTheEarth’, 669)	464	1256
(‘Yuuupthatsme’, ‘RT @Yuuupthatsme: TEGO SIĘ NIE SPODZIEWAŁAM XD\n#WeLoveTheEarth https://t.co/VuhRNdhcV7', 669)	70	316
(‘Yuuupthatsme’, ‘RT @Yuuupthatsme: Miałeś być żyrafą #WeLoveTheEarth https://t.co/0kNCpU8o6q', 669)	8	56
(’thremptyworlds’, ‘RT @thremptyworlds: piosenka jest cudowna, tylko brakuje mi takiego momentu w ktorym wszyscy razem zaspiewaliby refren… #WeLoveTheEarth’, 629)	55	215
(’typicalbizzzle’, ‘RT @typicalbizzzle: This song and music video had such a powerful and great message it. It’s a wake up call for the whole woltd to get the…’, 601)	694	1116
(’liveforshawn02’, ‘RT @liveforshawn02: This is one of the most beautiful concepts for a video ever! I’m so proud and thankful of @shawnmendes❤️ @lildickytweet…’, 578)	1	3
(‘aurelialmao’, ‘RT @aurelialmao: Uważam, że tą piosenkę powinni znać wszyscy, bo może wtedy kogoś ruszy stan naszej planety #EarthMusicVideo #WeLoveTheEarth’, 572)	1	2
(‘jaileystation’, ‘RT @jaileystation: PERO USTEDES ESCUCHARON A ED SHEERAN SIENDO EL KOALA MAS TIERNO QUE EXISTE #WeLoveTheEarth https://t.co/UvmXmy5E1C', 546)	332	852
(‘athenajasvier’, ‘RT @athenajasvier: lil dicky is amazing #WeLoveTheEarth’, 546)	2	0
(’llostinmyouth’, ‘RT @llostinmyouth: Ed versione koala è il mio stile di vita dal lunedì alla domenica #WeLoveTheEarth https://t.co/MbhaMaVaRD', 520)	11	34
(‘MercurySimons’, “RT @MercurySimons: everybody should be listening to this masterpiece, the planet really needs our help and it shouldn’t take a song with a…”, 516)	275	489
(‘realestbeach’, ‘RT @realestbeach: if y’all love the earth, start acting like it. pollution is real and y’all need to start taking it seriously!! #WeLoveTh…’, 504)	354	681
(‘Piatt_Futuro’, ‘RT @Piatt_Futuro: Edilizia motore di rigenerazione urbana. #4future #19aprile #fridaysforfuture ☀️ #WeLoveTheEarth\nhttps://t.co/TdVCrj86L8’, 427)	4	5
(‘Piatt_Futuro’, ‘RT @Piatt_Futuro: Il nostro “ius soli”:\xa0 terra, suolo e prodotti agricoli 🇮🇹 nel mondo. #4future #19aprile #fridaysforfuture ☀️ #WeLoveTheE…’, 427)	2	4
(‘Piatt_Futuro’, “RT @Piatt_Futuro: L’ambiente è la leva dello sviluppo economico. #4future #19aprile #fridaysforfuture\n#WeLoveTheEarth\nhttps://t.co/TdVCrj86…”, 427)	5	5
(‘Piatt_Futuro’, “RT @Piatt_Futuro: La persona al centro, l’ambiente intorno. #4future #19aprile ☀️ #fridaysforfuture #WeLoveTheEarth\nhttps://t.co/TdVCrj86L8”, 427)	4	6
(‘Piatt_Futuro’, ‘RT @Piatt_Futuro: Meno gomma, più ferro: trasporti veloci sicuri e non inquinanti. #4future #19aprile #fridaysforfuture ☀️ #WeLoveTheEarth…’, 427)	5	6
(‘Piatt_Futuro’, ‘RT @Piatt_Futuro: Riscaldamento globale: soluzioni, non fake news! #4future #19aprile #fridaysforfuture ☀️ #WeLoveTheEarth\nhttps://t.co/TdV…’, 427)	12	12
(‘Piatt_Futuro’, “RT @Piatt_Futuro: Più mercato e concorrenza per l’accesso di tutti\xa0 all’acqua, meno sprechi e inefficienze. #4future #19aprile #fridaysforf…”, 427)	1	1
(‘Piatt_Futuro’, “RT @Piatt_Futuro: Un grande sconto fiscale per l’ambiente! #4future #19aprile #fridaysforfuture ☀️ #WeLoveTheEarth\nhttps://t.co/TdVCrj86L8”, 427)	4	6
(‘Piatt_Futuro’, ‘RT @Piatt_Futuro: Riciclo riuso e termocombustori: Italia pulita, meno malavita. #4future #19aprile #fridaysforfuture ☀️\n#WeLoveTheEarth\nht…’, 427)	4	8
(‘vanessaibanez’, ‘RT @vanessaibanez: artists using their platform to spread awareness, kudos!! stream the song #WeLoveTheEarth https://t.co/nID21ksupN', 416)	1	1
(‘Adryelli_Godoi’, ‘RT @Adryelli_Godoi: Justin/ Babuíno: Nós vamos morrer?!\nLil Dicky: Sabe, Bieber nós podemos morrer. Eu não vou mentir pra você. Há muitas p…’, 387)	629	1095
(‘fayessflatline’, ‘RT @fayessflatline: This is some true shit #WeLoveTheEarth https://t.co/MLYY7v3JAL', 381)	8220	16514
(‘brian_prism’, ‘RT @brian_prism: #ConCalmaRemix & #WeLoveTheEarth\xa0 Tonight🔥 https://t.co/EjQbaYZYmu', 348)	16	22
(‘Clausbelieberj’, ‘RT @Clausbelieberj: progetto meraviglioso, vederlo mi ha emozionata.\ncoinvolgere così tanti artisti per un buona causa è una cosa bellissim…’, 337)	14	68
(‘ver0017’, ‘RT @ver0017: oni doskonale wiedzą, że handwritten przyczyniło się do uratowania naszej planety #WeLoveTheEarth https://t.co/XF0JvxrerA', 288)	9	42
(‘ver0017’, ‘RT @ver0017: ciekawi mnie na jakiej podstawie dobierali zwierzęta do artystów 😂🤔 #WeLoveTheEarth’, 288)	1	2
(‘dsakrv’, ‘RT @dsakrv: Lil Dicky - Earth (Official Music Video) https://t.co/M5TtrumWYQ via @YouTube #WeLoveTheEarth’, 239)	1	1
(‘CelebrityArticl’, ‘RT @CelebrityArticl: #ArianaGrande Opens Up About the Darkish Facet of Performing Her New #music “It Is Hell” #Singer #CelebrityNews #WeLov…’, 217)	3	2
(‘Ws_Vinicius’, ‘RT @Ws_Vinicius: O Brasil não foi esquecido no churrasco, Rio representando manooo, com os vocais da Ariana de fundo. QUE HINOOOOOOOO EU AM…’, 215)	224	516
(‘sevenyearsbiebs’, ‘RT @sevenyearsbiebs: oby wiadomość zawarta w piosence trafiła do dużej ilości osób, może dzięki temu świat zacznie się zmieniać na lepsze…’, 214)	546	1083
(‘ShawnDailyJP’, ‘RT @ShawnDailyJP: Lil Dickyの新曲・チャリティーソング「Earth」\nアリアナ・ジャスティン・カニエら名だたるアーティスト達と共にショーンも参加してます。\n是非聴いてみて下さい。\n#WeLoveTheEarth \nhttps://t.co/CKUtq0…’, 199)	1	6
(‘Joprojekt’, “RT @Joprojekt: #WeLoveTheEarth Dire que #LinkinPark a écrit deux albums magnifiques sur l’environnement (pas seulement) #AThousandSuns et #…”, 182)	1	3
(‘justinsmiIxs’, ‘RT @justinsmiIxs: “Ya sé que no todos somos iguales, pero vivimos en la misma tierra” \n”¿Vamos a morir?” \n"Sabes Bieber? No te voy a menti…’, 171)	530	762
(‘jujustinbieeber’, ‘RT @jujustinbieeber: Que la canción y el video no se quede solo un momento y creamos conciencia del daño que hacemos a los animales, al pla…’, 167)	115	178
(‘sgxjbstylesoft’, ‘RT @sgxjbstylesoft: la voz de todos esos artistas tan talentosos ha quedado perfecta, amo que se hayan juntando para dejar ese mensaje tan…’, 156)	2	1
(‘jestemzajeta’, ‘RT @jestemzajeta: uzbierane pieniądze trafiają na fundację Leonardo DiCaprio, która ratuje naszą planetę \n#WeLoveTheEarth’, 143)	504	1218
(‘jestemzajeta’, ‘RT @jestemzajeta: Shawn jest nosorożecem\nHalsey lwiątkiem\nCharlie Puth żyrafą\nEd Sheeran koalą\nSia kangurem \nmiley słoniem \nAriana zebrą \nK…’, 143)	126	408
(‘itszGabbs’, ‘RT @itszGabbs: OUÇAM EARTH! O clipe é o incrível, a música mais ainda e todo dinheiro arrecadado com a música será doado para a instituição…’, 105)	72	105
(‘Clem_rocher’, ‘RT @Clem_rocher: Justin Bieber, Ariana Grande, Wiz Khalifa, Shawn Mendes, Ed Sheeran, Rita Ora… de nombreux artistes se mobilisent autour…’, 99)	30	54
(‘OursPooh’, ‘RT @OursPooh: we love the earth, it is our planet\nwe love the earth, it is our home #WeLoveTheEarth https://t.co/NXUhU758uQ', 98)	85	263
(‘soulsfiend’, ‘RT @soulsfiend: mais les gens qui sont là à se plaindre «\xa0gngngn j’aime pas la chanson\xa0» vous pouvez connecter vos 2 neurones 2 secondes ?…’, 82)	24	35
(‘introandrew’, ‘RT @introandrew: #WeLoveTheEarth é un progetto molto importante,oltre ad essere una canzone,vuole lanciare un messaggio,il SALVARE LA TERRA…’, 81)	53	119
(’twentysevenmoon’, ‘RT @twentysevenmoon: i recommend you guys go watch Our Planet on netflix. it made me cry and i hope it motivates you to take action on savi…’, 75)	106	274
(‘camillazambett2’, ‘RT @camillazambett2: Spero che con questa collaborazione arrivi a tutti il messaggio di salvare la Terra!🌍❤️\n #WeLoveTheEarth https://t.co/…’, 74)	3	0
(‘bemybizzle’, “RT @bemybizzle: Justin and Ariana they both sounds so damn good it’s just wow !!!!\n#WeLoveTheEarth https://t.co/9aRwrD6i9J", 67)	762	1911
(‘xjbtalentedx’, ‘RT @xjbtalentedx: Yo podría ver y ver éste vídeo nunca me aburriría… ♡\n\n#WeLoveTheEarth https://t.co/HdLb55CNcA', 58)	63	171
(’alfredo30’, ‘RT @alfredo30: Questa canzone è bellissima.\nSperiamo riesca a sensibilizzare più persone possibili su questo enorme pericolo 🌍\n#WeLoveT…’, 55)	4	7
(‘protectmayne’, ‘RT @protectmayne: Earth se tornou minha música preferida junto com o clipe \nisso simplesmente por passar uma mensagem forte pra gente, eu e…’, 40)	294	562
(‘fabiannn29’, ‘RT @fabiannn29: Me encantaría tanto que esta canción fuera #1 no solo en USA o UK si no en muchos países del mundo. Ojalá que le den el…’, 37)	5	8
(‘4miraizat’, ‘RT @4miraizat: ed sheeran(and koala) is my spirit animal #WeLoveTheEarth https://t.co/VYTPuAajM6', 28)	68	181
(‘LILG96958980’, ‘RT @LILG96958980: Me on my way to save the earth after bobbing to the song for a straight hour \n#WeLoveTheEarth https://t.co/94g0N2SLBM', 24)	1224	3448
(‘KeyNarumi’, ‘RT @KeyNarumi: Ini keren sumpah. Pesannya tuh bener bikin lo mikir apa aja yg udah lo lakuin untuk Bumi Kita dengan visual yang menyenangka…’, 18)	1	0
(‘Yuberli_Rosario’, ‘RT @Yuberli_Rosario: “Todos estos tiroteos, la contaminación. Nos atacamos a nosotros mismos. Vamos a relajarnos, respeta lo que construimo…’, 17)	24	54
(‘bieber_najaFC’, ‘RT @bieber_najaFC: Com o coração quentinho depois de ver que realmente tem pessoas que se importam com o nosso planeta. #WeLoveTheEarth htt…’, 14)	26	69
(‘khaliqhaiqal’, ‘RT @khaliqhaiqal: Marvel : “Avengers Endgame is gonna be the most ambitious crossover event in history”\n\nLil Dicky : hold my beer\n@lildi…’, 13)	2151	5487
(‘martyydg’, ‘RT @martyydg: Semplicemente MERAVIGLIOSO #WeLoveTheEarth https://t.co/m6I2J2soEe', 12)	2	2
(‘biebermylife946’, ‘RT @biebermylife946: Salviamo il posto in cui viviamo,acquistando la canzone daremo una speranza. #WeLoveTheEarth https://t.co/6OmM8oAvhG', 12)	2	2
(‘Malikjvvd’, ‘RT @Malikjvvd: Uwuuu uwuuuuuu\n#WeLoveTheEarth https://t.co/WtVzUrVJW1', 1)	64	208

9. Analyzing used languages

🕵️‍♀️ Our table tells us that:

Lil Dicky's followers reacted the most — 42.4% of his followers liked his first tweet.
Even if celebrities like Katy Perry and Ellen have a huuge Twitter following, their followers hardly reacted, e.g., only 0.0098% of Katy's followers liked her tweet.
While Leo got the most likes and retweets in terms of counts, his first tweet was only liked by 2.19% of his followers.

The large differences in reactions could be explained by the fact that this was Lil Dicky's music video. Leo still got more traction than Katy or Ellen because he played some major role in this initiative.

Can we find some more interesting patterns in the data? From the text of the tweets, we could spot different languages, so let's create a frequency distribution for the languages.

# Extracting language for each tweet and appending it to the list of languages
tweets_languages = []
for tweet in tweets:
 tweets_languages.append(tweet['lang'])

# Plotting the distribution of languages
%matplotlib inline
plt.hist(tweets_languages)

Expand Output

 (array([303., 107., 22., 14., 36., 32., 3., 2., 1., 2.]),
 array([ 0. , 1.3, 2.6, 3.9, 5.2, 6.5, 7.8, 9.1, 10.4, 11.7, 13. ]),
 <BarContainer object of 10 artists>)

10. Final thoughts

🕵️‍♀️ The last histogram tells us that:

Most of the tweets were in English.
Polish, Italian and Spanish were the next runner-ups.
There were a lot of tweets with a language alien to Twitter (lang = 'und').

Why is this sort of information useful? Because it can allow us to get an understanding of the "category" of people interested in this topic (clustering).

What an exciting journey it has been! We started almost clueless, and here we are.. rich in insights.

From location based comparisons to analyzing the activity around a tweet to finding patterns from languages and devices, we have covered a lot today — let's give ourselves a well-deserved pat on the back! ✋

Magic Formula = Data + Python + Creativity + Curiosity

"Does owning an Apple compared to Andorid influences people's propensity towards this trend?"

From the looks of the current tweet trends, we see more tweets from iPhone than Android.

device_os = []
for tweet in tweets:
 if 'android' in tweet['source']:
 os = 'Android'
 else:
 os = 'iPhone'

 device_os.append(os)

# Plotting the distribution of languages
%matplotlib inline
d = Counter(device_os)
plt.bar(d.keys(),d.values())

Book Recommendations engine from Charles Darwin

Wed, 28 Sep 2022 13:04:04 +0000

1. Darwin’s bibliography

Charles Darwin is one of the few universal figures of science. His most renowned work is without a doubt his "On the Origin of Species" published in 1859 which introduced the concept of natural selection. But Darwin wrote many other books on a wide range of topics, including geology, plants or his personal life. In this notebook, we will automatically detect how closely related his books are to each other.

To this purpose, we will develop the bases of a content-based book recommendation system, which will determine which books are close to each other based on how similar the discussed topics are. The methods we will use are commonly used in text- or documents-heavy industries such as legal, tech or customer support to perform some common task such as text classification or handling search engine queries.

Let's take a look at the books we'll use in our recommendation system.

# Import library
import glob

# The books files are contained in this folder
folder = "datasets/"

# List all the .txt files and sort them alphabetically
files = glob.glob(folder+'*.txt',recursive=True)

files

`` ['datasets/Autobiography.txt', 'datasets/CoralReefs.txt', 'datasets/DescentofMan.txt', 'datasets/DifferentFormsofFlowers.txt', 'datasets/EffectsCrossSelfFertilization.txt', 'datasets/ExpressionofEmotionManAnimals.txt', 'datasets/FormationVegetableMould.txt', 'datasets/FoundationsOriginofSpecies.txt', 'datasets/GeologicalObservationsSouthAmerica.txt', 'datasets/InsectivorousPlants.txt', 'datasets/LifeandLettersVol1.txt', 'datasets/LifeandLettersVol2.txt', 'datasets/MonographCirripedia.txt', 'datasets/MonographCirripediaVol2.txt', 'datasets/MovementClimbingPlants.txt', 'datasets/OriginofSpecies.txt', 'datasets/PowerMovementPlants.txt', 'datasets/VariationPlantsAnimalsDomestication.txt', 'datasets/VolcanicIslands.txt', 'datasets/VoyageBeagle.txt'] ``

2. Load the contents of each book into Python

As a first step, we need to load the content of these books into Python and do some basic pre-processing to facilitate the downstream analyses. We call such a collection of texts a corpus. We will also store the titles for these books for future reference and print their respective length to get a gauge for their contents.

# Import libraries
import re, os

# Initialize the object that will contain the texts and titles
txts = []
titles = []
T = True

for n in files:
 # Open each file
 f = open(n, encoding='utf-8-sig')
 # Remove all non-alpha-numeric characters
 filtered = re.sub('[\W_]+', ' ', f.read())
 # Store the texts and titles of the books in two separate lists
 txts.append(filtered)
 titles.append(os.path.basename(n).replace(".txt", ""))

# Print the length, in characters, of each book
[len(t) for t in txts]

`` [123231, 496068, 1776539, 617088, 913713, 624232, 335920, 523021, 797401, 901406, 1047518, 1010643, 767492, 1660866, 298319, 916267, 1093567, 1043499, 341447, 1149574] ``

3. Find “On the Origin of Species”

For the next parts of this analysis, we will often check the results returned by our method for a given book. For consistency, we will refer to Darwin's most famous book: "On the Origin of Species." Let's find to which index this book is associated.

# Browse the list containing all the titles
# print(titles[:2])
ori = 0
for i in range(len(titles)):
 # Store the index if the title is "OriginofSpecies"
 if titles[i] == 'OriginofSpecies':
 ori = i

# Print the stored index
print(ori)

 15
```

## 4. Tokenize the corpus
<p>As a next step, we need to transform the corpus into a format that is easier to deal with for the downstream analyses. We will tokenize our corpus, i.e., transform each text into a list of the individual words (called tokens) it is made of. To check the output of our process, we will print the first 20 tokens of "<em>On the Origin of Species</em>".</p>


```python
# Define a list of stop words
stoplist = set('for a of the and to in to be which some is at that we i who whom show via may my our might as well'.split())

# Convert the text to lower case
txts_lower_case = [txt.lower() for txt in txts]


# Transform the text into tokens
txts_split = [txt.split() for txt in txts_lower_case]

# Remove tokens which are part of the list of stop words
texts = []
for txts in txts_split:
 temp = []
 for token in txts:
 if token not in stoplist:
 temp.append(token)
 texts.append(temp)

# Print the first 20 tokens for the "On the Origin of Species" book
print(texts[ori][:20])

`` ['on', 'origin', 'species', 'but', 'with', 'regard', 'material', 'world', 'can', 'least', 'go', 'so', 'far', 'this', 'can', 'perceive', 'events', 'are', 'brought', 'about'] ``

5. Stemming of the tokenized corpus

If you have read On the Origin of Species, you will have noticed that Charles Darwin can use different words to refer to a similar concept. For example, the concept of selection can be described by words such as selection, selective, select or selects. This will dilute the weight given to this concept in the book and potentially bias the results of the analysis.

To solve this issue, it is a common practice to use a stemming process, which will group together the inflected forms of a word so they can be analysed as a single item: the stem. In our On the Origin of Species example, the words related to the concept of selection would be gathered under the select stem.

As we are analysing 20 full books, the stemming algorithm can take several minutes to run and, in order to make the process faster, we will directly load the final results from a pickle file and review the method used to generate it.

import pickle

# Load the Porter stemming function from the nltk package
# from nltk.stem import PorterStemmer

# Create an instance of a PorterStemmer object
# porter = PorterStemmer()

# For each token of each text, we generated its stem 
# texts_stem = [[porter.stem(token) for token in text] for text in texts]

# Save to pickle file
# pickle.dump(texts_stem, open("datasets/texts_stem.p", "wb" ))

# Load the stemmed tokens list from the pregenerated pickle file
texts_stem = pickle.load(open('datasets/texts_stem.p', 'rb'))

# Print the 20 first stemmed tokens from the "On the Origin of Species" book
print(texts_stem[ori][:20])

`` ['on', 'origin', 'speci', 'but', 'with', 'regard', 'materi', 'world', 'can', 'least', 'go', 'so', 'far', 'thi', 'can', 'perceiv', 'event', 'are', 'brought', 'about'] ``

6. Building a bag-of-words model

Now that we have transformed the texts into stemmed tokens, we need to build models that will be useable by downstream algorithms.

First, we need to will create a universe of all words contained in our corpus of Charles Darwin's books, which we call a dictionary. Then, using the stemmed tokens and the dictionary, we will create bag-of-words models (BoW) of each of our texts. The BoW models will represent our books as a list of all uniques tokens they contain associated with their respective number of occurrences.

To better understand the structure of such a model, we will print the five first elements of one of the "On the Origin of Species" BoW model.

# Load the functions allowing to create and use dictionaries
from gensim import corpora

# Create a dictionary from the stemmed tokens
dictionary = corpora.Dictionary()

# Create a bag-of-words model for each book, using the previously generated dictionary
bows = [dictionary.doc2bow(doc,allow_update=True) for doc in texts_stem]

# Print the first five elements of the On the Origin of species' BoW model
bows[ori][:5]

`` [(0, 11), (5, 51), (6, 1), (8, 2), (21, 1)] ``

7. The most common words of a given book

The results returned by the bag-of-words model is certainly easy to use for a computer but hard to interpret for a human. It is not straightforward to understand which stemmed tokens are present in a given book from Charles Darwin, and how many occurrences we can find.

In order to better understand how the model has been generated and visualize its content, we will transform it into a DataFrame and display the 10 most common stems for the book "On the Origin of Species".

# Import pandas to create and manipulate DataFrames
import pandas as pd

# Convert the BoW model for "On the Origin of Species" into a DataFrame
df_bow_origin = pd.DataFrame(bows[ori])

# Add the column names to the DataFrame
df_bow_origin.columns = ['index','occurrences']

# Add a column containing the token corresponding to the dictionary index
df_bow_origin['token'] = [dictionary[idx] for idx in df_bow_origin['index']]

# Sort the DataFrame by descending number of occurrences and print the first 10 values
df_bow_origin.sort_values('occurrences',ascending = False).head(10)

	index	occurrences	token
748	1168	2023	have
1119	1736	1558	on
1489	2288	1543	speci
892	1366	1480	it
239	393	1362	by
1128	1747	1201	or
125	218	1140	are
665	1043	1137	from
1774	2703	1000	with
1609	2452	962	thi

8. Build a tf-idf model

If it wasn't for the presence of the stem "speci", we would have a hard time to guess this BoW model comes from the On the Origin of Species book. The most recurring words are, apart from few exceptions, very common and unlikely to carry any information peculiar to the given book. We need to use an additional step in order to determine which tokens are the most specific to a book.

To do so, we will use a tf-idf model (term frequency–inverse document frequency). This model defines the importance of each word depending on how frequent it is in this text and how infrequent it is in all the other documents. As a result, a high tf-idf score for a word will indicate that this word is specific to this text.

After computing those scores, we will print the 10 words most specific to the "On the Origin of Species" book (i.e., the 10 words with the highest tf-idf score).

# Load the gensim functions that will allow us to generate tf-idf models
from gensim.models import TfidfModel

# Generate the tf-idf model
model = TfidfModel(bows)

# Print the model for "On the Origin of Species"
model[bows[ori]]

`` [(8, 0.00020383224047642202), (21, 0.0005716037746542094), (23, 0.0017118699041370883), (27, 0.0006458270601429994), (28, 0.0025678048562056324), (31, 0.0008559349520685442), (35, 0.00101497410751472), (36, 0.00101497410751472), (51, 0.000886740665721021), (54, 0.00202994821502944), (56, 0.0023757190244598344), (57, 0.00010191612023821101), (63, 0.0027544680933525786), (64, 0.000509580601191055), (66, 0.00020383224047642202), (67, 0.0023757190244598344), (68, 0.00202994821502944), (75, 0.0013772340466762893), (76, 0.0004433703328605105), (78, 0.004171843479607349), (80, 0.0020859217398036746), (83, 0.00857405661981314), (84, 0.000509580601191055), (88, 0.002445986885717064), (89, 0.0033632319678609636), (90, 0.000886740665721021), (91, 0.0016747506839411234), (94, 0.000886740665721021), (95, 0.0004433703328605105), (96, 0.003546962662884084), (97, 0.0016306579238113761), (102, 0.037686478293143394), (104, 0.000917245082143899), (106, 0.001417375386254771), (108, 0.0035434384656369273), (109, 0.005299638252386972), (111, 0.0022421546452406423), (114, 0.0015287418035731652), (123, 0.0509769226270304), (125, 0.009580115302391834), (126, 0.004171843479607349), (127, 0.001417375386254771), (137, 0.020026648174904797), (139, 0.007749924721715993), (141, 0.00101916120238211), (143, 0.004751438048919669), (144, 0.0047597336465067894), (154, 0.0010467191774632021), (156, 0.013962508472634907), (165, 0.006781184131501494), (167, 0.000611496721429266), (172, 0.021771758891234606), (176, 0.0033632319678609636), (178, 0.0031035923300235736), (186, 0.009274366941677202), (188, 0.0016747506839411234), (192, 0.006257765219411025), (196, 0.0038749623608579963), (197, 0.0013772340466762893), (198, 0.000886740665721021), (204, 0.0042521261587643135), (207, 0.0022603947105004976), (212, 0.001324909563096743), (214, 0.020395035311583484), (215, 0.021720943436859957), (219, 0.0019374811804289981), (220, 0.004396220545345449), (221, 0.0054618131386103995), (222, 0.0018206043795367997), (223, 0.0007086876931273855), (224, 0.007703414568616897), (226, 0.0030574836071463303), (230, 0.005135609712411265), (231, 0.0027544680933525786), (235, 0.002445986885717064), (236, 0.0012916541202859988), (237, 0.0020934383549264042), (241, 0.0029308136968969663), (242, 0.0015865778821689297), (243, 0.008263404280057736), (245, 0.004520789421000995), (246, 0.003465148088099174), (247, 0.001732574044049587), (249, 0.0018840945194337638), (251, 0.008343686959214698), (252, 0.0030449223225441605), (253, 0.0006458270601429994), (261, 0.001324909563096743), (269, 0.0006458270601429994), (271, 0.0008373753419705617), (276, 0.009921627703783398), (278, 0.0034296226479252566), (280, 0.0021260630793821567), (283, 0.02138147122013851), (285, 0.001417375386254771), (287, 0.0013301109985815313), (288, 0.006886170233381447), (290, 0.0035434384656369273), (291, 0.0018206043795367997), (296, 0.009839160268154101), (298, 0.042694255446964965), (300, 0.0016747506839411234), (301, 0.000611496721429266), (302, 0.0042521261587643135), (303, 0.004171843479607349), (304, 0.0020859217398036746), (311, 0.0016306579238113761), (313, 0.007086876931273855), (323, 0.0047597336465067894), (325, 0.021606217490500734), (327, 0.001732574044049587), (329, 0.01790404260679176), (335, 0.0004433703328605105), (336, 0.0023922081541910096), (338, 0.0020859217398036746), (339, 0.001417375386254771), (344, 0.001417375386254771), (345, 0.011527628654373272), (346, 0.00020383224047642202), (348, 0.0006280315064779214), (349, 0.0006458270601429994), (351, 0.0036412087590735995), (354, 0.007980665991489189), (356, 0.0018840945194337638), (358, 0.013772340466762893), (359, 0.0022603947105004976), (362, 0.00101497410751472), (367, 0.0023922081541910096), (369, 0.19772097392783367), (370, 0.043694505108883196), (371, 0.000509580601191055), (372, 0.0023922081541910096), (373, 0.0016306579238113761), (374, 0.001732574044049587), (375, 0.001936406284526009), (376, 0.006489658900271853), (377, 0.0030449223225441605), (380, 0.0016145676503574987), (387, 0.0007086876931273855), (388, 0.0042521261587643135), (389, 0.0005716037746542094), (391, 0.0023922081541910096), (400, 0.00202994821502944), (406, 0.013772340466762893), (407, 0.0005716037746542094), (409, 0.0035588452033748874), (411, 0.0007086876931273855), (412, 0.004751438048919669), (418, 0.0027544680933525786), (421, 0.0027544680933525786), (424, 0.007538884401734597), (425, 0.00811979286011776), (426, 0.0032291353007149973), (429, 0.000886740665721021), (431, 0.000886740665721021), (432, 0.010149741075147201), (433, 0.0016306579238113761), (434, 0.00405989643005888), (436, 0.0021260630793821567), (442, 0.009212940010656012), (446, 0.004171843479607349), (448, 0.021562415055741965), (449, 0.03291890683694216), (450, 0.0035434384656369273), (453, 0.0011210773226203211), (454, 0.00405989643005888), (456, 0.011301973552502492), (457, 0.0020859217398036746), (458, 0.0003229135300714997), (463, 0.01790404260679176), (464, 0.0027544680933525786), (465, 0.0008559349520685442), (468, 0.00202994821502944), (470, 0.0018206043795367997), (478, 0.010923626277220799), (482, 0.0344479258546676), (484, 0.013772340466762893), (486, 0.0020859217398036746), (489, 0.0054618131386103995), (490, 0.022603947105004983), (491, 0.00405989643005888), (493, 0.0007086876931273855), (497, 0.012839024281028162), (498, 0.0035434384656369273), (502, 0.00637818923814647), (505, 0.011970998987233783), (507, 0.0009687405902144991), (514, 0.0025121260259116855), (520, 0.004280477050004863), (524, 0.0020859217398036746), (526, 0.005716037746542095), (527, 0.0027544680933525786), (529, 0.006318799454769082), (531, 0.003546962662884084), (532, 0.012791353704852355), (534, 0.005320443994326125), (536, 0.0014268256833349542), (541, 0.0011432075493084189), (543, 0.01110604517518251), (544, 0.0031401575323896065), (546, 0.0017148113239626283), (551, 0.004784416308382019), (552, 0.0031731557643378595), (558, 0.0023922081541910096), (559, 0.00020383224047642202), (561, 0.0022421546452406423), (562, 0.005197722132148762), (563, 0.006908346571257135), (564, 0.0011432075493084189), (565, 0.00405989643005888), (566, 0.0013772340466762893), (569, 0.04034670029030646), (573, 0.010630315396910783), (576, 0.0025121260259116855), (578, 0.0028580188732710474), (579, 0.003159399727384541), (582, 0.0035434384656369273), (586, 0.003563578536689751), (590, 0.0017118699041370883), (592, 0.0010467191774632021), (593, 0.0018206043795367997), (594, 0.015898914757160917), (596, 0.0015865778821689297), (598, 0.0023922081541910096), (600, 0.01251553043882205), (601, 0.00202994821502944), (604, 0.05582357591330961), (605, 0.00637818923814647), (606, 0.0005716037746542094), (616, 0.0010467191774632021), (619, 0.00710481875260304), (620, 0.024945049756828264), (626, 0.0013772340466762893), (628, 0.00041868767098528084), (629, 0.004197875890929496), (633, 0.0013301109985815313), (635, 0.004960813851891699), (636, 0.005991544664479809), (641, 0.006346311528675719), (646, 0.0027544680933525786), (649, 0.005763814327186636), (652, 0.000917245082143899), (653, 0.0011432075493084189), (654, 0.0023757190244598344), (655, 0.0016747506839411234), (657, 0.0030449223225441605), (658, 0.007104097661572994), (660, 0.005144433971887885), (662, 0.003563578536689751), (663, 0.0011878595122299172), (665, 0.00101497410751472), (666, 0.006395676852426178), (667, 0.01623958572023552), (668, 0.005669501545019084), (670, 0.011723254787587865), (674, 0.002445986885717064), (675, 0.0450089246309177), (676, 0.061655829302082535), (678, 0.00040766448095284403), (679, 0.005144433971887885), (681, 0.0007134128416674771), (682, 0.0015865778821689297), (685, 0.0020859217398036746), (686, 0.0006458270601429994), (693, 0.0035670642083373855), (698, 0.0022864150986168378), (705, 0.0034237398082741766), (712, 0.0017118699041370883), (713, 0.0013772340466762893), (720, 0.0029308136968969663), (722, 0.005074870537573601), (726, 0.01771719232818464), (727, 0.004396220545345449), (728, 0.0031731557643378595), (729, 0.006089844645088321), (730, 0.01028886794377577), (731, 0.0017148113239626283), (740, 0.008357121859533303), (741, 0.0066990027357644935), (743, 0.006930296176198348), (744, 0.01033323296228799), (745, 0.006257765219411025), (748, 0.017254559827750243), (750, 0.07127157073379503), (752, 0.1459896647842414), (753, 0.06980942681543291), (758, 0.004197875890929496), (769, 0.0017118699041370883), (770, 0.0010467191774632021), (771, 0.0007134128416674771), (776, 0.00020934383549264042), (783, 0.006346311528675719), (784, 0.004171843479607349), (785, 0.002834750772509542), (786, 0.004171843479607349), (787, 0.006930296176198348), (788, 0.002955567486908119), (789, 0.003546962662884084), (793, 0.014208195323145987), (797, 0.004197875890929496), (798, 0.024359378580353284), (803, 0.010149741075147201), (805, 0.0034237398082741766), (806, 0.02571547930590961), (810, 0.0011878595122299172), (811, 0.006886170233381447), (815, 0.0018840945194337638), (816, 0.0012560630129558428), (817, 0.0015865778821689297), (819, 0.0026602219971630626), (820, 0.006458270601429995), (821, 0.006207184660047147), (822, 0.0688958517093352), (825, 0.0003229135300714997), (830, 0.001222993442858532), (831, 0.003552048830786497), (832, 0.02256933073236843), (833, 0.01097906002243099), (834, 0.0017118699041370883), (838, 0.001936406284526009), (839, 0.0016306579238113761), (842, 0.0013772340466762893), (848, 0.006886170233381447), (849, 0.0011432075493084189), (851, 0.0020859217398036746), (857, 0.0005716037746542094), (859, 0.008343686959214698), (861, 0.0018840945194337638), (862, 0.001732574044049587), (866, 0.004433703328605105), (867, 0.004784416308382019), (870, 0.0003229135300714997), (873, 0.01886292456358891), (875, 0.0009687405902144991), (878, 0.00202994821502944), (879, 0.005489530011215495), (881, 0.0007086876931273855), (883, 0.0008153289619056881), (885, 0.0023922081541910096), (887, 0.001732574044049587), (891, 0.0006458270601429994), (892, 0.0013772340466762893), (894, 0.002445986885717064), (897, 0.005991544664479809), (898, 0.0011432075493084189), (903, 0.0011878595122299172), (905, 0.0025678048562056324), (909, 0.002649819126193486), (910, 0.005508936186705157), (917, 0.05877026247301295), (919, 0.0017118699041370883), (921, 0.0011210773226203211), (924, 0.0054015543726251836), (925, 0.004784416308382019), (929, 0.013718490591701027), (931, 0.00101497410751472), (935, 0.0026602219971630626), (936, 0.0025121260259116855), (937, 0.0009687405902144991), (939, 0.0016306579238113761), (944, 0.000886740665721021), (945, 0.00710481875260304), (948, 0.0023757190244598344), (951, 0.012692623057351438), (952, 0.006089844645088321), (953, 0.005074870537573601), (956, 0.010149741075147201), (957, 0.0012560630129558428), (958, 0.0008559349520685442), (962, 0.0011432075493084189), (966, 0.12753430785821307), (967, 0.047703783053191846), (971, 0.0019374811804289981), (973, 0.000509580601191055), (975, 0.00040766448095284403), (976, 0.00203832240476422), (980, 0.009502876097839338), (981, 0.016526808560115472), (982, 0.00101497410751472), (985, 0.03886905667648624), (988, 0.004382393170243073), (992, 0.0014268256833349542), (994, 0.007932889410844648), (995, 0.021725146310165012), (996, 0.011527628654373272), (997, 0.0022603947105004976), (998, 0.03014551231094022), (999, 0.009519467293013579), (1000, 0.00010191612023821101), (1004, 0.0011878595122299172), ...] ``

9. The results of the tf-idf model

Once again, the format of those results is hard to interpret for a human. Therefore, we will transform it into a more readable version and display the 10 most specific words for the "On the Origin of Species" book.

# Convert the tf-idf model for "On the Origin of Species" into a DataFrame
df_tfidf = pd.DataFrame(model[bows[ori]])

# Name the columns of the DataFrame id and score
df_tfidf.columns = ['id','score']

# Add the tokens corresponding to the numerical indices for better readability
df_tfidf['token'] = [dictionary[idx] for idx in df_tfidf.id]

# Sort the DataFrame by descending tf-idf score and print the first 10 rows.
df_tfidf.sort_values('score',ascending = False).head(10)

	id	score	token
878	2164	0.327414	select
3106	10108	0.203908	pigeon
128	369	0.197721	breed
2988	9396	0.167496	migrat
945	2325	0.148186	steril
284	752	0.145990	domest
503	1255	0.128272	hybrid
370	966	0.127534	fertil
3889	16210	0.124392	rtner
3540	12715	0.121197	naturalis

10. Compute distance between texts

The results of the tf-idf algorithm now return stemmed tokens which are specific to each book. We can, for example, see that topics such as selection, breeding or domestication are defining "On the Origin of Species" (and yes, in this book, Charles Darwin talks quite a lot about pigeons too). Now that we have a model associating tokens to how specific they are to each book, we can measure how related to books are between each other.

To this purpose, we will use a measure of similarity called cosine similarity and we will visualize the results as a distance matrix, i.e., a matrix showing all pairwise distances between Darwin's books.

# Load the library allowing similarity computations
from gensim import similarities

# Compute the similarity matrix (pairwise distance between all texts)
sims = similarities.MatrixSimilarity(model[bows])

# Transform the resulting list into a dataframe
sim_df = pd.DataFrame(list(sims))

# Add the titles of the books as columns and index of the dataframe
sim_df.index = titles
sim_df.columns = titles

# Print the resulting matrix
sim_df

	Autobiography	CoralReefs	DescentofMan	DifferentFormsofFlowers	EffectsCrossSelfFertilization	ExpressionofEmotionManAnimals	FormationVegetableMould	FoundationsOriginofSpecies	GeologicalObservationsSouthAmerica	InsectivorousPlants	LifeandLettersVol1	LifeandLettersVol2	MonographCirripedia	MonographCirripediaVol2	MovementClimbingPlants	OriginofSpecies	PowerMovementPlants	VariationPlantsAnimalsDomestication	VolcanicIslands	VoyageBeagle
Autobiography	1.000000	0.049467	0.080428	0.066482	0.077184	0.088723	0.040678	0.059271	0.030562	0.014878	0.396709	0.217129	0.005686	0.008483	0.022856	0.099991	0.016247	0.049018	0.038556	0.183507
CoralReefs	0.049467	1.000000	0.009480	0.001952	0.001923	0.004999	0.029432	0.022096	0.061027	0.002276	0.030965	0.017558	0.006324	0.010579	0.001518	0.039089	0.002707	0.011586	0.057514	0.267749
DescentofMan	0.080428	0.009480	1.000000	0.072761	0.029968	0.148670	0.027055	0.135775	0.009698	0.009404	0.059684	0.080314	0.053506	0.043275	0.005146	0.267554	0.011357	0.232841	0.007882	0.123917
DifferentFormsofFlowers	0.066482	0.001952	0.072761	1.000000	0.391834	0.006474	0.010585	0.040104	0.002846	0.007502	0.015933	0.046523	0.009405	0.005484	0.008151	0.128909	0.018964	0.050023	0.002611	0.013124
EffectsCrossSelfFertilization	0.077184	0.001923	0.029968	0.391834	1.000000	0.006844	0.032262	0.040288	0.002246	0.006777	0.019504	0.046504	0.003212	0.002962	0.014932	0.146441	0.039770	0.055132	0.002178	0.017140
ExpressionofEmotionManAnimals	0.088723	0.004999	0.148670	0.006474	0.006844	1.000000	0.020985	0.047202	0.005217	0.011475	0.064873	0.048886	0.016825	0.029897	0.005913	0.062979	0.011317	0.083847	0.005561	0.098961
FormationVegetableMould	0.040678	0.029432	0.027055	0.010585	0.032262	0.020985	1.000000	0.021470	0.067989	0.035589	0.027916	0.023620	0.019866	0.023984	0.038820	0.049259	0.040182	0.033147	0.059407	0.097908
FoundationsOriginofSpecies	0.059271	0.022096	0.135775	0.040104	0.040288	0.047202	0.021470	1.000000	0.028028	0.006023	0.057820	0.054782	0.007618	0.010883	0.003973	0.322405	0.008788	0.194533	0.017590	0.089132
GeologicalObservationsSouthAmerica	0.030562	0.061027	0.009698	0.002846	0.002246	0.005217	0.067989	0.028028	1.000000	0.006879	0.028551	0.012104	0.009687	0.024738	0.002043	0.058046	0.003491	0.014389	0.373249	0.260141
InsectivorousPlants	0.014878	0.002276	0.009404	0.007502	0.006777	0.011475	0.035589	0.006023	0.006879	1.000000	0.005967	0.016518	0.019214	0.020023	0.249814	0.014961	0.023056	0.010522	0.008544	0.014776
LifeandLettersVol1	0.396709	0.030965	0.059684	0.015933	0.019504	0.064873	0.027916	0.057820	0.028551	0.005967	1.000000	0.885828	0.005752	0.012772	0.005388	0.097457	0.009505	0.055259	0.026374	0.171708
LifeandLettersVol2	0.217129	0.017558	0.080314	0.046523	0.046504	0.048886	0.023620	0.054782	0.012104	0.016518	0.885828	1.000000	0.004967	0.010843	0.017565	0.096955	0.012099	0.050764	0.011806	0.089947
MonographCirripedia	0.005686	0.006324	0.053506	0.009405	0.003212	0.016825	0.019866	0.007618	0.009687	0.019214	0.005752	0.004967	1.000001	0.522273	0.012441	0.029902	0.018694	0.023460	0.010754	0.014342
MonographCirripediaVol2	0.008483	0.010579	0.043275	0.005484	0.002962	0.029897	0.023984	0.010883	0.024738	0.020023	0.012772	0.010843	0.522273	1.000000	0.006802	0.036755	0.022376	0.030669	0.017952	0.025047
MovementClimbingPlants	0.022856	0.001518	0.005146	0.008151	0.014932	0.005913	0.038820	0.003973	0.002043	0.249814	0.005388	0.017565	0.012441	0.006802	1.000000	0.008802	0.104966	0.011530	0.002832	0.012282
OriginofSpecies	0.099991	0.039089	0.267554	0.128909	0.146441	0.062979	0.049259	0.322405	0.058046	0.014961	0.097457	0.096955	0.029902	0.036755	0.008802	1.000000	0.018266	0.405333	0.036014	0.164661
PowerMovementPlants	0.016247	0.002707	0.011357	0.018964	0.039770	0.011317	0.040182	0.008788	0.003491	0.023056	0.009505	0.012099	0.018694	0.022376	0.104966	0.018266	1.000000	0.020589	0.003819	0.024149
VariationPlantsAnimalsDomestication	0.049018	0.011586	0.232841	0.050023	0.055132	0.083847	0.033147	0.194533	0.014389	0.010522	0.055259	0.050764	0.023460	0.030669	0.011530	0.405333	0.020589	1.000000	0.012620	0.114134
VolcanicIslands	0.038556	0.057514	0.007882	0.002611	0.002178	0.005561	0.059407	0.017590	0.373249	0.008544	0.026374	0.011806	0.010754	0.017952	0.002832	0.036014	0.003819	0.012620	1.000000	0.138323
VoyageBeagle	0.183507	0.267749	0.123917	0.013124	0.017140	0.098961	0.097908	0.089132	0.260141	0.014776	0.171708	0.089947	0.014342	0.025047	0.012282	0.164661	0.024149	0.114134	0.138323	1.000000

11. The book most similar to “On the Origin of Species”

We now have a matrix containing all the similarity measures between any pair of books from Charles Darwin! We can now use this matrix to quickly extract the information we need, i.e., the distance between one book and one or several others.

As a first step, we will display which books are the most similar to "On the Origin of Species," more specifically we will produce a bar chart showing all books ranked by how similar they are to Darwin's landmark work.

# This is needed to display plots in a notebook
%matplotlib inline

# Import libraries
import matplotlib.pyplot as plt

# Select the column corresponding to "On the Origin of Species" and 
v = sim_df.iloc[:,ori]

# Sort by ascending scores
v_sorted = v.sort_values()
v_sorted

# Plot this data has a horizontal bar plot
v_sorted.plot(kind = 'barh')

# Modify the axes labels and plot title for a better readability
plt.xlabel('Score')
plt.ylabel('Book')
plt.title("Similarity with book \'On the Origin of Species\'")

<matplotlib.text.Text at 0x7f65c9dbdfd0>

12. Which books have similar content?

This turns out to be extremely useful if we want to determine a given book's most similar work. For example, we have just seen that if you enjoyed "On the Origin of Species," you can read books discussing similar concepts such as "The Variation of Animals and Plants under Domestication" or "The Descent of Man, and Selection in Relation to Sex." If you are familiar with Darwin's work, these suggestions will likely seem natural to you. Indeed, On the Origin of Species has a whole chapter about domestication and The Descent of Man, and Selection in Relation to Sex applies the theory of natural selection to human evolution. Hence, the results make sense.

However, we now want to have a better understanding of the big picture and see how Darwin's books are generally related to each other (in terms of topics discussed). To this purpose, we will represent the whole similarity matrix as a dendrogram, which is a standard tool to display such data. This last approach will display all the information about book similarities at once. For example, we can find a book's closest relative but, also, we can visualize which groups of books have similar topics (e.g., the cluster about Charles Darwin personal life with his autobiography and letters). If you are familiar with Darwin's bibliography, the results should not surprise you too much, which indicates the method gives good results. Otherwise, next time you read one of the author's book, you will know which other books to read next in order to learn more about the topics it addressed.

# Import libraries
from scipy.cluster import hierarchy

# Compute the clusters from the similarity matrix,
# using the Ward variance minimization algorithm
Z = hierarchy.linkage(sim_df.values, method = 'ward')

# Display this result as a horizontal dendrogram
plot = hierarchy.dendrogram(Z,leaf_font_size=8, labels=sim_df.index, orientation="left")

Dataset is taken from DataCamp

Customer Segmentation Report

Wed, 14 Sep 2022 09:37:32 +0000

Customer segmentation is a way to identify groups of similar customers. Customers can be segmented on a wide variety of characteristics, such as demographic information, purchase behaviour, and attitudes. This template provides an end-to-end report for processing and segmenting customer purchase data using a K-means clustering algorithm. It also includes a snake plot and heatmap to visualize the resulting clusters and feature importance.

Multiple numerical variables that you can use for clustering.
No NaN/NA values. You can use this to impute missing values if needed.

The dataset consists of customer data, including purchase recency, frequency, and monetary value. Each row represents a different customer with a distinct customer ID.

1. Loading packages and Inspecting the Data

The code below imports the packages necessary for data manipulation, visualization, pre-processing, and clustering. It also sets up the visualization style and loads in the data.

Finally, it inspects the data types and missing values with the .info() method from pandas.

# Load packages
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans

# Set visualization style
sns.set_style("darkgrid")

# Load the data and replace with your CSV file path
df = pd.read_csv("data/customer_data.csv")

# Preview the data
df

	CustomerID	Recency	Frequency	MonetaryValue
0	12747	3	25	948.70
1	12748	1	888	7046.16
2	12749	4	37	813.45
3	12820	4	17	268.02
4	12822	71	9	146.15
...	...	...	...	...
3638	18280	278	2	38.70
3639	18281	181	2	31.80
3640	18282	8	2	30.70
3641	18283	4	152	432.93
3642	18287	43	15	395.76

3643 rows × 4 columns

# Check columns for data types and missing values
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3643 entries, 0 to 3642
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 CustomerID 3643 non-null int64
1 Recency 3643 non-null int64
2 Frequency 3643 non-null int64
3 MonetaryValue 3643 non-null float64
dtypes: float64(1), int64(3)
memory usage: 114.0 KB

2. Exploring the Data

Based on the evaluation above, you can select columns you wish to inspect further. In this template, three columns are selected from the four columns. CustomerID is omitted because it is an identifier and not useful for clustering.

The code below reduces the DataFrame to the columns you wish to cluster on and then prints descriptive statistics using the describe() method from pandas.

Printing descriptive statistics is helpful because K-means clustering has several key assumptions that can be revealed via this exploration:

There is no skewness to the data.
The variables have the same average values.
The variables have the same variance.

If you’d like to learn more about pre-processing data for K-means clustering, you can refer to this video from the course Customer Segmentation in Python.

# Select columns for clustering
columns_for_clustering = ["Recency", "Frequency", "MonetaryValue"]

# Create new DataFrame with clustering variables
df_features = df[columns_for_clustering]

# Print a summary of descriptive statistics
df_features.describe()

	Recency	Frequency	MonetaryValue
count	3643.00000	3643.000000	3643.000000
mean	90.43563	18.714247	370.694387
std	94.44651	43.754468	1347.443451
min	1.00000	1.000000	0.650000
25%	19.00000	4.000000	58.705000
50%	51.00000	9.000000	136.370000
75%	139.00000	21.000000	334.350000
max	365.00000	1497.000000	48060.350000

The facetgrid() function from seaborn creates a grid of histograms of the data to be clustered. It serves as a further exploration of the data to determine its skew and whether it needs transformation.

# Plot the distributions of the selected variables
g = sns.FacetGrid(
 df_features.melt(), # Reformat the DataFrame for plotting purposes
 col="variable", # Split on the 'variable' column created by reformating
 sharey=False, # Turn off shared y-axis
 sharex=False, # Turn off shared x-axis
)
# Apply a histogram to the facet grid
g.map(sns.histplot, "value")
# Adjust the top of the plots to make room for the title
g.fig.subplots_adjust(top=0.8)
# Create a title
g.fig.suptitle("Unprocessed Variable Distributions", fontsize=16)
plt.show()

Before proceeding, it is crucial to ensure that all columns selected for clustering are numeric. The following code iterates through the reduced DataFrame and checks whether each column is numeric. If it returns True, then you can proceed with the pre-processing.

all([pd.api.types.is_numeric_dtype(df_features[col]) for col in columns_for_clustering])

True

3. Pre-processing the Data

Based on the grids above, if there is a skew, you will have to complete this step which removes the skew and center the variables. This is the case for the placeholder dataset used in this template and will likely be the case for your data.

First, a log transformation is applied to the data using the numpy log() function. A log transformation unskews the data in preparation for clustering.
Next, the StandardScaler() from sklearn.preprocessing fits and transforms the log-transformed data. This centers and scales the data in further preparation for clustering.
Finally, a new DataFrame is created and visualized again to confirm the results.

# Perform a log transformation of the data to unskew the data
df_log = np.log(df_features)

# Initialize a standard scaler and fit it
scaler = StandardScaler()
scaler.fit(df_log)

# Scale and center the data
df_normalized = scaler.transform(df_log)

# Create a pandas DataFrame of the processed data
df_processed = pd.DataFrame(
 data=df_normalized, index=df_features.index, columns=df_features.columns
)

# Plot the distributions of the selected variables
g = sns.FacetGrid(df_processed.melt(), col="variable")
g.map(sns.histplot, "value")
g.fig.subplots_adjust(top=0.8)
g.fig.suptitle("Preprocessed Variable Distributions", fontsize=16)
plt.show()

4. Choosing the Number of Clusters

The next step is to fit a variable number of clusters and plot each cluster’s sum-of-squared errors (SSE). The SSE reflects the sum of squared distances from every data point to the cluster center. The aim is to reduce the SSE while still maintaining a reasonable number of clusters.

By plotting the SSE for each number of clusters, you can identify at what point there are diminishing returns by adding new clusters. This type of plot is called an elbow plot.

In the code below, you can set the maximum number of clusters you want to plot, and then a loop is used to generate the SSE for each number of clusters. Finally, the seaborn function pointplot() plots a curve with each cluster number and SSE. This allows you to identify the ’elbow’ or point where there are only marginal reductions for each additional cluster.

# Set the maximum number of clusters to plot
max_clusters = 10

# Initialize empty dictionary to store sum of squared errors
sse = {}

# Fit KMeans and calculate SSE for each k
for k in range(1, max_clusters):
 # Initialize KMeans with k clusters
 kmeans = KMeans(n_clusters=k, random_state=1)
 # Fit KMeans on the normalized dataset
 kmeans.fit(df_processed)
 # Assign sum of squared distances to k element of dictionary
 sse[k] = kmeans.inertia_

# Initialize a figure of set size
plt.figure(figsize=(10, 4))

# Create an elbow plot of SSE values for each key in the dictionary
sns.pointplot(x=list(sse.keys()), y=list(sse.values()))

# Add labels to the plot
plt.title("Elbow Method Plot", fontsize=16) # Add a title to the plot
plt.xlabel("Number of Clusters") # Add x-axis label
plt.ylabel("SSE") # Add y-axis label

# Show the plot
plt.show()

5. Clustering the Data

You can now select an optimal number of clusters based on the elbow plot above by setting k. In this example, k is set to 3.

KMeans() from sklearn.cluster with k clusters is then fit to the processed data, and the cluster labels are extracted and assigned back to the original data. This allows you to inspect raw data by cluster in later steps.

# Choose number of clusters
k = 3

# Initialize KMeans
kmeans = KMeans(n_clusters=k, random_state=1)

# Fit k-means clustering on the normalized data set
kmeans.fit(df_processed)

# Extract cluster labels
cluster_labels = kmeans.labels_

# Create a new DataFrame by adding a new cluster column to the original data
df_clustered = df.assign(Cluster=cluster_labels)

# Preview the clustered DataFrame
df_clustered

	CustomerID	Recency	Frequency	MonetaryValue	Cluster
0	12747	3	25	948.70	0
1	12748	1	888	7046.16	0
2	12749	4	37	813.45	0
3	12820	4	17	268.02	0
4	12822	71	9	146.15	2
...	...	...	...	...	...
3638	18280	278	2	38.70	1
3639	18281	181	2	31.80	1
3640	18282	8	2	30.70	1
3641	18283	4	152	432.93	0
3642	18287	43	15	395.76	2

3643 rows × 5 columns

6. Inspecting the Clusters

6a. Visualizing the Raw Values by Cluster

The next step is to analyze the unprocessed data by cluster. The pandas method DataFrame.groupby(), combined with the .size() method, returns the total number of rows per Cluster.

# Group the data by cluster and calculate the total number of rows per group
df_sizes = df_clustered.groupby(["Cluster"], as_index=False).size()

# Inspect the row counts
df_sizes

	Cluster	size
0	0	901
1	1	1156
2	2	1586

Next, the mean values per cluster are visualized. The data is grouped again, and this time, the pandas method .mean() is used to aggregate the data by cluster and calculate the mean for each variable. Alternatively, the .agg() method can also be used to specify specific aggregations for different columns if necessary. Consult the documentation for further information on the types of aggregations possible.

The seaborn catplot() function visualizes the means per cluster.

# Calculate the mean of feature columns by cluster
df_means = df_clustered.groupby(["Cluster"])[df_features.columns].mean().reset_index()

# Plot the distributions of the selected variables
sns.catplot(
 data=df_means.melt(id_vars="Cluster"), # Transform the data to enable plotting
 col="variable",
 x="Cluster",
 y="value",
 kind="bar",
)

# Add a title
plt.suptitle("Average Values by Cluster", y=1.04, fontsize=16)

# Show the plot
plt.show()

6b. Create a Snake Plot of the Clusters

The next step takes the processed data and visualizes the differences between the clusters using a snake plot. This can be helpful in spotting trends or key differences that would not be visible with the raw data.

# Assign cluster labels to processed DataFrame
df_processed_clustered = df_processed.assign(Cluster=cluster_labels)

# Melt the normalized DataFrame and reset the index
df_processed_melt = pd.melt(
 df_processed_clustered.reset_index(),
 # Assign the cluster labels as the ID
 id_vars=['Cluster'],
 # Assign clustering variables as values
 value_vars=df_features.columns,
 # Name the variable and value
 var_name="Metric",
 value_name="Value",
)

# Change the figure size
plt.figure(figsize=(10, 6))

# Add label and titles to the plot
plt.title('Snake Plot of Normalized Variables', fontsize=16)
plt.xlabel('Metric')
plt.ylabel('Average Normalized Value')

# Plot a line for each value of the cluster variable
sns.lineplot(data=df_processed_melt, x='Metric', y='Value', hue='Cluster')
plt.show()

6c. Create a Heatmap of Relative Importance

Another technique to help visualize how each segment is distinct is to plot the relative importance. The code below achieves this by doing the following:

First, it calculates the average values for each cluster.
Next, it calculates the average values for the total population.
It then divides the cluster averages by the population averages and subtracts one.

This provides a relative importance score for each of the different features used for clustering. The seaborn heatmap() function plots these relative importances on a red-to-blue colour scale to help visualize the relative importance of each attribute to the segments.

# Calculate average RFM values for each cluster
cluster_avg = df_clustered.groupby(["Cluster"])[columns_for_clustering].mean()

# Calculate average RFM values for the total customer population
population_avg = df[columns_for_clustering].mean()

# Calculate relative importance of cluster's attribute value compared to the population
relative_imp = cluster_avg / population_avg - 1

# Change the figure size
plt.figure(figsize=(8, 4))

# Add the plot title
plt.title("Relative importance of attributes", fontsize=16)

# Plot the heatmap
sns.heatmap(data=relative_imp, annot=True, fmt=".2f", cmap=sns.diverging_palette(20, 220, as_cmap=True))
plt.show()

This concludes the report! Hope you find it enjoyable & insightful.