HW5 -- Artificial Artificial Intelligence¶
In [1]:
import pandas as pd
import numpy as np
neg = pd.read_csv('AMT_neg.csv')
pos = pd.read_csv('AMT_pos.csv')
Initial EDA¶
In [5]:
neg[:3]
Out[5]:
In [6]:
pos[:3]
Out[6]:
In [7]:
neg.columns.tolist()
Out[7]:
How many unique turkers worked on each dataframe?¶
In [31]:
def get_unique(df, column):
unique = np.unique(df[column], return_counts=True)
df = pd.DataFrame(zip(unique[0], unique[1]))
return len(unique[0]), unique, df
num_neg, unique_neg, u_neg_df = get_unique(neg, 'WorkerId')
num_pos, unique_pos, u_pos_df = get_unique(pos, 'WorkerId')
print(num_neg, 'Turkers worked on NEG batch')
print(num_pos, 'Turkers worked on POS batch')
How many HITS did each unique turker do?¶
In [32]:
u_neg_df.plot(kind='bar',x=0,y=1)
Out[32]:
In [33]:
u_pos_df.plot(kind='bar',x=0,y=1)
Out[33]:
What's the max and min HIT for unique turkers¶
In [39]:
print('For {}, the min was: {} and the max was: {}'.format('neg', unique_neg[1].min(), unique_neg[1].max()))
print('For {}, the min was: {} and the max was: {}'.format('pos', unique_pos[1].min(), unique_pos[1].max()))
Did a specitic Sentiment take longer for turkers to assess?¶
In [20]:
import seaborn as sns
import matplotlib.pyplot as plt
sns.catplot(x="Answer.sentiment.label",
y="WorkTimeInSeconds",
kind="bar",
order=['Negative', 'Neutral', 'Positive'],
data=neg);
plt.title('Negative')
Out[20]:
In [19]:
sns.catplot(x="Answer.sentiment.label",
y="WorkTimeInSeconds",
kind="bar",
order=['Negative', 'Neutral', 'Positive'],
data=pos)
plt.title('Positive')
Out[19]:
How many turkers had less than 10 second response time?¶
In [44]:
response_time = neg[neg['WorkTimeInSeconds'] < 10]
response_time_check = neg[neg['WorkTimeInSeconds'] > 10]
In [45]:
len(response_time)
Out[45]:
In [46]:
len(response_time_check)
Out[46]:
Checking for potential bots¶
Did anyone have a consistent average low response time?¶
In [74]:
count = pos.groupby(['WorkerId'])['HITId'].count()
work_time = pos.groupby(['WorkerId'])['WorkTimeInSeconds'].mean()
new_df = pd.DataFrame([work_time, count]).T
new_df[:5]
Out[74]:
Did anyone have a consistent average high response time?¶
In [75]:
new_df['WorkTimeInMin'] = new_df['WorkTimeInSeconds']/60
new_df[:5]
Out[75]:
In [86]:
count = pos.groupby(['WorkerId', 'Answer.sentiment.label'])['Answer.sentiment.label'].count()
# count = pos.groupby(['WorkerId'])['Answer.sentiment.label'].count()
count
Out[86]:
Did anyone answer ONLY pos/neg/neutral?¶
In [117]:
pnn = pd.DataFrame()
pnn['Neutral'] = pos.groupby('WorkerId')['Answer.sentiment.label'].apply(lambda x: (x=='Neutral').sum())
pnn['Positive'] = pos.groupby('WorkerId')['Answer.sentiment.label'].apply(lambda x: (x=='Positive').sum())
pnn['Negative'] = pos.groupby('WorkerId')['Answer.sentiment.label'].apply(lambda x: (x=='Negative').sum())
pnn['Total'] = pos.groupby('WorkerId')['Answer.sentiment.label'].apply(lambda x: x.count())
pnn[:5]
Out[117]:
This is getting a little confusing, let's just look at our top performers¶
In [122]:
top = pnn.sort_values(by=['Total'], ascending=False)
In [123]:
top[:10]
Out[123]:
Interesting!! Looking from here, we have three workers who ONLY chose positive.
Let's look at their response time to see if we can determine if they are a bot!!
In [130]:
top['Avg_WorkTimeInSeconds'] = pos.groupby('WorkerId')['WorkTimeInSeconds'].apply(lambda x: x.mean())
top['Avg_WorkTimeInMin'] = pos.groupby('WorkerId')['WorkTimeInSeconds'].apply(lambda x: x.mean()/60)
top['Min_WorkTimeInMin'] = pos.groupby('WorkerId')['WorkTimeInSeconds'].apply(lambda x: x.min()/60)
top['Max_WorkTimeInMin'] = pos.groupby('WorkerId')['WorkTimeInSeconds'].apply(lambda x: x.max()/60)
In [131]:
top[:10]
Out[131]:
Even more interesting! These two don't appear to be bots, based on our current metric which is time variability.
HOWEVER, worker A681XM15AN28F appears to only work for an average of 13 seconds per review which doesn't seem like enough time to read and judge a review...
PART 2: Second submission to AMT¶
TOO MANY REVIEWERS!
Here is when we realized that doing a kappa score with over 30 individual reviewers would be tricky, so we rusubmitted to AMT and required the turkers to be 'Master' in the hopes that this additional barrier-to-entry would help reduce the amount of turkers working on the project
In [138]:
v2 = pd.read_csv('HW5_amt_v2.csv')
v2[:5]
len(v2)
Out[138]:
This time, I didn't separate the df into pos and neg before submitting to AMT, so we have to reimport the labels.
In [136]:
labels = pd.read_csv('all_JK_extremes_labeled.csv')
In [139]:
len(labels)
Out[139]:
Oops! That's right, we replicated each review * 3 so three separate people could look at each review
In [160]:
labels2 = labels.append([labels] * 2, ignore_index=True)
In [161]:
len(labels2)
Out[161]:
In [162]:
labels2.sort_values(by='0')
Out[162]:
Shoot! I realized I had to delete some emojis for the csv to be accepted by AMT, so the reviews themselves won't actually be matching... solution: Create two 'for-matching' columns made up of the first 5 words of each review
In [171]:
v2['for_matching'] = v2.apply(lambda x: x['Input.text'].split()[:5], axis=1)
In [172]:
labels2['for_matching'] = labels2.apply(lambda x: x['0'].split()[:5], axis=1)
Annnnnd why did I do that when I could just sort the df and apply the PoN
In [176]:
sorted_labels = labels2.sort_values(by='0')
sorted_labels[:6]
Out[176]:
In [226]:
sorted_v2 = v2.sort_values(by='Input.text')
sorted_v2[sorted_v2.columns[-5:]][:6]
Out[226]:
In [192]:
all_df = sorted_v2.copy()
# all_df['PoN'] = sorted_labels['PoN'].tolist()
# THIS DIDN'T WORK BECAUSE I DIDN'T WAIT UNTIL ALL WERE DONE FROM AMT. RESEARCHER ERROR BUT OMG I HATE MYSELF
In [193]:
len(all_df)
Out[193]:
In [194]:
293/3
Out[194]:
Confirming that YEP. 293 isn't divisible by 3, meaning I didn't wait until the last turker finished. omg.
Reuploading now -- WITH BETTER CODE AND BETTER VARIABLE NAMES!¶
In [224]:
turker = pd.read_csv('HW5_amt_294.csv')
print(len(turker))
turker[turker.columns[-5:]][:5]
Out[224]:
In [197]:
# Getting labels...
labels = pd.read_csv('all_JK_extremes_labeled.csv')
# X3
labels = labels.append([labels] * 2, ignore_index=True)
print(len(labels))
labels[:5]
Out[197]:
NOW, TO SORT!¶
In [198]:
sorted_labels = labels.sort_values(by=['0'])
sorted_turker = turker.sort_values(by=['Input.text'])
In [199]:
sorted_labels[:5]
Out[199]:
In [206]:
sorted_turker['Input.text'][:5]
Out[206]:
OMG HOORAY HOORAY HOORAY!!
NOTE: FUN FACT!! I can type here and then hit the esc key to turn this cell into markdown!!
In [223]:
# YUCK THIS IS SO AGGRIVATING!! This line below doens't work because it still uses indexes.
# So the P and N didn't match up
# sorted_turker['PoN'] = sorted_labels['PoN']
sorted_turker['PoN'] = sorted_labels['PoN'].tolist()
sorted_turker[sorted_turker.columns[-5:]][:5]
Out[223]:
PART 3: ANALYZE¶
First, let's clean ALL the things
In [237]:
all_df = sorted_turker[['Input.text', 'WorkerId', 'Answer.sentiment.label', 'PoN']]
In [238]:
all_df[:5]
Out[238]:
In [242]:
all_df_all = all_df.copy()
all_df_all['APoN'] = all_df_all.apply(lambda x: x['Answer.sentiment.label'][0], axis=1)
In [243]:
all_df_all
Out[243]:
In [244]:
all_df_all['agree'] = all_df_all.apply(lambda x: x['PoN'] == x['APoN'], axis=1)
In [263]:
all_df_all[-10:]
Out[263]:
Lets see how many agree!¶
In [361]:
gdf = pd.DataFrame(all_df_all.groupby(['Input.text','PoN'])['agree'].mean())
gdf_forplot = gdf.copy()
gdf
Out[361]:
OK so this actually gave us something we want... BUT PLEASE TELL ME THE BETTER WAY!!
In [362]:
three_agreed = gdf[gdf['agree'] == 1]
len(three_agreed)
Out[362]:
In [363]:
three_agreed_but_wrong = gdf[gdf['agree'] == 0]
len(three_agreed_but_wrong)
Out[363]:
In [364]:
disparity = gdf[(gdf['agree'] > 0) & (gdf['agree'] < 1)]
len(disparity)
Out[364]:
In [365]:
quickdf = pd.DataFrame({'labels': ['agreed', 'agreed, incorrect', 'disparity'], 'counts': [33,31,34]})
quickdf
Out[365]:
In [366]:
quickdf.plot(kind='bar', x='labels', y='counts')
Out[366]:
Lol that is not super useful
In [367]:
# # three_agreed = gdf[(gdf['agree'] == 1) & (gdf['PoN'] == 'P')]
# # len(three_agreed)
# # gdf['PoN']
# gdf = gdf.reset_index()
# p_three_agreed = gdf[(gdf['agree'] == 1) & (gdf['PoN'] == 'P')]
# len(p_three_agreed)
In [368]:
# gdf = gdf.reset_index(drop=True)
# n_three_agreed = gdf[(gdf['agree'] == 1) & (gdf['PoN'] == 'N')]
# len(n_three_agreed)
In [369]:
# gdf = gdf.reset_index(drop=True)
# p_three_agreed_wrong = gdf[(gdf['agree'] == 0) & (gdf['PoN'] == 'P')]
# len(p_three_agreed_wrong)
In [370]:
# gdf = gdf.reset_index(drop=True)
# n_three_agreed_wrong = gdf[(gdf['agree'] == 0) & (gdf['PoN'] == 'N')]
# len(n_three_agreed_wrong)
In [371]:
# gdf = gdf.reset_index(drop=True)
# p_disparity = gdf[((gdf['agree'] / 1) != 0 ) & (gdf['PoN'] == 'P')]
# len(p_disparity)
In [372]:
# gdf = gdf.reset_index(drop=True)
# n_disparity = gdf[(gdf['agree'] / 1 != 0 ) & (gdf['PoN'] == 'N')]
# len(n_disparity)
In [373]:
# quickdf = pd.DataFrame({'labels': ['positive', 'negative'], 'counts': [18,15]})
# quickdf.plot(kind='bar', x='labels', y='counts')
In [374]:
# sns.catplot(x="Answer.sentiment.label",
# y="WorkTimeInSeconds",
# kind="bar",
# order=['Negative', 'Neutral', 'Positive'],
# data=gdp_forplot);
# plt.title('By Polarity')
gdf_forplot = gdf_forplot.reset_index()
In [380]:
gdf_forplot.groupby(['agree'])['agree'].count()
Out[380]:
In [382]:
gdf_forplot.groupby(['agree','PoN'])['agree'].count()
Out[382]:
In [ ]:
In [ ]: