HOW TO SUMMARIZE IN PYTHON

Following this tutorial! | 10-13-19

STEP 1: GET THE DATA!!

Step 1a: Import libraries

# file = open('../summary_test_positive.txt').readlines()
file = open('../summary_test_negative.txt').readlines()

file

['bad . bad . \n',
 'bad . \n',
 'that one word seems to pretty much sums up beyond the valley of the dolls . \n',
 "if that summary isn't enough for you , how about t&a , t&a , t&a ? \n",
 "still haven't got the point ? \n",
 "other than director russ meyer's predilection for casting attractive large breasted women who ultimately expose the afore-mentioned anatomical areas , there is really only one other reason to recommend even taking a look at this movie . \n",
 'that is the fact that it was co-written by famed film critic roger ebert , who also was responsible for the screenplay . \n',
 'after watching this movie you will never be able to sit through another one of his reviews where he gives a movie a thumbs down for bad writing with a straight face . \n',
 'this movie stinks out loud . \n',
 'quite frankly , this movie deserves a . \n',
 'but there are parts of it that are so bad they are almost funny . \n',
 "so i'm giving it a . \n",
 'and maybe that is too generous . \n',
 'right from the opening credits , i knew that i had a class-a bomb on my hands . \n',
 'not only are the way the credits actually shot distracting , but the first scene you see includes a big breasted young woman being chased by a guy in a nazi uniform . \n',
 'i had absolutely no idea why the hell that was happening ( it does get explained later ) and as soon as the first scene is over , we cut to a completely unrelated scene . \n',
 'to be honest , as i sat through this movie mesmerized by just how incredibly awful it was , i actually forgot about the seemingly out of place opening until it popped up again later in the film . \n',
 "with the quality of the writing during the rest of the film , it wouldn't have surprised me if the opening had never been explained . \n",
 'so what is this movie about ? \n',
 'you ask . \n',
 'like it really matters . \n',
 'ok , here goes . \n',
 'this all-girl band headed by kelly macnamara ( dolly reed ) and her friends go to hollywood to try to gain a foothold in the music industry . \n',
 "once there , they do manage to find success ( due as much to their hooters as anything else ? it sure wasn't for their brutally bad singing voices ) , and the movie chronicles how their lives change for the worse as the pressures of fame get to them . \n",
 'everything from big egos , to booze and drugs to free flowing sex sends them on a downward spiral . \n',
 'there are a couple of other idiotic subplots thrown in for good measure , but the fame is the one that pretty much sums up this thing . \n',
 '>from a creative standpoint there is nothing redeeming here . \n',
 'other than the above-mentioned obsession with big knockers that russ meyer seemed to have . \n',
 'the dialogue is so incredibly bad that it literally is funny in parts . \n',
 'mr . ebert has generously thrown in helpings of " hey man " , " dig " and my all time favorite -- " this is my happening , and it freaks me out " . \n',
 'now i ask you , with lines like that how can you go wrong ? \n',
 'ebert had tried to inject as many big words as possible into the dialogue . \n',
 'maybe he thought it would make the movie seem smarter . \n',
 "i don't know , but all the big words in the world wouldn't be able to disguise the bad writing and even worse acting . \n",
 'but the wretched dialogue goes along well with the wretched quality of everything else in this movie . \n',
 "i've seen home movies directed better than meyer managed with this turkey . \n",
 'in fact , there is one scene -- the one in which they are in a van driving to hollywood to make their fortunes -- during which i really had to question if meyer or his editors had just suffered serious head injuries . \n',
 'add to the directing and writing the music in this movie . \n',
 'i almost got up to check my sound system to see if it was broken , there was such a pile of crap emanating from the speakers . \n',
 'then we have the cast . \n',
 'first lets start david gurian who played harris , the manager of the band . \n',
 'this has got to be the goofiest looking guy that has ever set foot in front of a motion picture camera . \n',
 "sadly , his acting doesn't come close to making up for his looks . \n",
 "if you have been following along up to this point , this shouldn't surprise you . \n",
 "meyer's stable of well endowed girls also have the benefit of being fairly attractive to go along with their other assets . \n",
 'dolly reed plays kelly , the leader of the band . \n',
 'and no surprise here , she was cast for her cup size , not her talents . \n',
 'and yes , she does loose the shirt a few times and display her impressive talents . \n',
 'sadly , her ass is almost as large as her chest . \n',
 "hey , it a sexist movie , so i'm writing a sexist review . \n",
 'then we have former playboy playmate cynthia myers in a fairly small role as casey , one of the other band members . \n',
 'this goes along with the rest of the idiotic thinking in the movie . \n',
 'meyer casts a gorgeous playmate with a rack to kill for and who obviously has no acting talent at all , but her nude scenes are the biggest disappointment of all . \n',
 'sure russ , now is the time to get artsy and throw in some well placed shadows . \n',
 'on the up side , she does have a fun lesbo scene . \n',
 "i sound like i'm writing a review in a porn magazine . \n",
 "but hey , i'll admit it ; the only reason that i actually managed to sit through this damn movie was to catch a look at cynthia myers naked . \n",
 'and since that was a huge disappointment , i pretty much wasted two hours of my life on this turkey . \n',
 'the only thing that i can say about this movie is that you should stay away from it . \n',
 'unless of course you want to feel good about yourself by knowing that even a pulitzer prize winning film critic like roger ebert has screwed up at least once in his life too . \n',
 "and if you are thinking of checking it out for the double d's -- you are better off just downloading nude cynthia myers pictures off the internet . \n",
 'this is a movie that should be avoided at all costs . \n',
 'an even better idea might be to require video stores to place a warning on the box of beyond the valley of the dolls -- beware : this movie is extremely hazardous to your common sense . \n',
 'proceed with extreme caution . ']

article_text = ""
for line in file:
    article_text += line.strip()
article_text[:1000]

"bad . bad .bad .that one word seems to pretty much sums up beyond the valley of the dolls .if that summary isn't enough for you , how about t&a , t&a , t&a ?still haven't got the point ?other than director russ meyer's predilection for casting attractive large breasted women who ultimately expose the afore-mentioned anatomical areas , there is really only one other reason to recommend even taking a look at this movie .that is the fact that it was co-written by famed film critic roger ebert , who also was responsible for the screenplay .after watching this movie you will never be able to sit through another one of his reviews where he gives a movie a thumbs down for bad writing with a straight face .this movie stinks out loud .quite frankly , this movie deserves a .but there are parts of it that are so bad they are almost funny .so i'm giving it a .and maybe that is too generous .right from the opening credits , i knew that i had a class-a bomb on my hands .not only are the way the cred"

STEP 2: CLEAN (& preprocess) THE DATA!!

Step 2a: Use regex and re.sub to remove square brackets and extra spaces from ORIGINAL article_text

import re
article_text = re.sub(r'\[[0-9]*\]', '', article_text)
article_text = re.sub(r'\s+', ' ', article_text)
article_text[:1000]

"bad . bad .bad .that one word seems to pretty much sums up beyond the valley of the dolls .if that summary isn't enough for you , how about t&a , t&a , t&a ?still haven't got the point ?other than director russ meyer's predilection for casting attractive large breasted women who ultimately expose the afore-mentioned anatomical areas , there is really only one other reason to recommend even taking a look at this movie .that is the fact that it was co-written by famed film critic roger ebert , who also was responsible for the screenplay .after watching this movie you will never be able to sit through another one of his reviews where he gives a movie a thumbs down for bad writing with a straight face .this movie stinks out loud .quite frankly , this movie deserves a .but there are parts of it that are so bad they are almost funny .so i'm giving it a .and maybe that is too generous .right from the opening credits , i knew that i had a class-a bomb on my hands .not only are the way the cred"

Step 2b: Use regex and re.sub to remove extra characters and digits for a new FORMATTED_TEXT variable

formatted_article_text = re.sub('[^a-zA-Z]', ' ', article_text)
formatted_article_text = re.sub(r'\s+', ' ', formatted_article_text)

formatted_article_text[:1000]

'bad bad bad that one word seems to pretty much sums up beyond the valley of the dolls if that summary isn t enough for you how about t a t a t a still haven t got the point other than director russ meyer s predilection for casting attractive large breasted women who ultimately expose the afore mentioned anatomical areas there is really only one other reason to recommend even taking a look at this movie that is the fact that it was co written by famed film critic roger ebert who also was responsible for the screenplay after watching this movie you will never be able to sit through another one of his reviews where he gives a movie a thumbs down for bad writing with a straight face this movie stinks out loud quite frankly this movie deserves a but there are parts of it that are so bad they are almost funny so i m giving it a and maybe that is too generous right from the opening credits i knew that i had a class a bomb on my hands not only are the way the credits actually shot distracting '

STEP 3: TOKENIZE SENTENCES!!

import nltk
sentence_list = nltk.sent_tokenize(article_text)
sentence_list[:5]

['bad .',
 "bad .bad .that one word seems to pretty much sums up beyond the valley of the dolls .if that summary isn't enough for you , how about t&a , t&a , t&a ?still haven't got the point ?other than director russ meyer's predilection for casting attractive large breasted women who ultimately expose the afore-mentioned anatomical areas , there is really only one other reason to recommend even taking a look at this movie .that is the fact that it was co-written by famed film critic roger ebert , who also was responsible for the screenplay .after watching this movie you will never be able to sit through another one of his reviews where he gives a movie a thumbs down for bad writing with a straight face .this movie stinks out loud .quite frankly , this movie deserves a .but there are parts of it that are so bad they are almost funny .so i'm giving it a .and maybe that is too generous .right from the opening credits , i knew that i had a class-a bomb on my hands .not only are the way the credits actually shot distracting , but the first scene you see includes a big breasted young woman being chased by a guy in a nazi uniform .i had absolutely no idea why the hell that was happening ( it does get explained later ) and as soon as the first scene is over , we cut to a completely unrelated scene .to be honest , as i sat through this movie mesmerized by just how incredibly awful it was , i actually forgot about the seemingly out of place opening until it popped up again later in the film .with the quality of the writing during the rest of the film , it wouldn't have surprised me if the opening had never been explained .so what is this movie about ?you ask .like it really matters .ok , here goes .this all-girl band headed by kelly macnamara ( dolly reed ) and her friends go to hollywood to try to gain a foothold in the music industry .once there , they do manage to find success ( due as much to their hooters as anything else ?",
 "it sure wasn't for their brutally bad singing voices ) , and the movie chronicles how their lives change for the worse as the pressures of fame get to them .everything from big egos , to booze and drugs to free flowing sex sends them on a downward spiral .there are a couple of other idiotic subplots thrown in for good measure , but the fame is the one that pretty much sums up this thing .>from a creative standpoint there is nothing redeeming here .other than the above-mentioned obsession with big knockers that russ meyer seemed to have .the dialogue is so incredibly bad that it literally is funny in parts .mr .",
 'ebert has generously thrown in helpings of " hey man " , " dig " and my all time favorite -- " this is my happening , and it freaks me out " .now i ask you , with lines like that how can you go wrong ?ebert had tried to inject as many big words as possible into the dialogue .maybe he thought it would make the movie seem smarter .i don\'t know , but all the big words in the world wouldn\'t be able to disguise the bad writing and even worse acting .but the wretched dialogue goes along well with the wretched quality of everything else in this movie .i\'ve seen home movies directed better than meyer managed with this turkey .in fact , there is one scene -- the one in which they are in a van driving to hollywood to make their fortunes -- during which i really had to question if meyer or his editors had just suffered serious head injuries .add to the directing and writing the music in this movie .i almost got up to check my sound system to see if it was broken , there was such a pile of crap emanating from the speakers .then we have the cast .first lets start david gurian who played harris , the manager of the band .this has got to be the goofiest looking guy that has ever set foot in front of a motion picture camera .sadly , his acting doesn\'t come close to making up for his looks .if you have been following along up to this point , this shouldn\'t surprise you .meyer\'s stable of well endowed girls also have the benefit of being fairly attractive to go along with their other assets .dolly reed plays kelly , the leader of the band .and no surprise here , she was cast for her cup size , not her talents .and yes , she does loose the shirt a few times and display her impressive talents .sadly , her ass is almost as large as her chest .hey , it a sexist movie , so i\'m writing a sexist review .then we have former playboy playmate cynthia myers in a fairly small role as casey , one of the other band members .this goes along with the rest of the idiotic thinking in the movie .meyer casts a gorgeous playmate with a rack to kill for and who obviously has no acting talent at all , but her nude scenes are the biggest disappointment of all .sure russ , now is the time to get artsy and throw in some well placed shadows .on the up side , she does have a fun lesbo scene .i sound like i\'m writing a review in a porn magazine .but hey , i\'ll admit it ; the only reason that i actually managed to sit through this damn movie was to catch a look at cynthia myers naked .and since that was a huge disappointment , i pretty much wasted two hours of my life on this turkey .the only thing that i can say about this movie is that you should stay away from it .unless of course you want to feel good about yourself by knowing that even a pulitzer prize winning film critic like roger ebert has screwed up at least once in his life too .and if you are thinking of checking it out for the double d\'s -- you are better off just downloading nude cynthia myers pictures off the internet .this is a movie that should be avoided at all costs .an even better idea might be to require video stores to place a warning on the box of beyond the valley of the dolls -- beware : this movie is extremely hazardous to your common sense .proceed with extreme caution .']

STEP 4: FIND WORD FREQUENCY, WEIGHTED!!

Step 4a: Remove Stopwords

stopwords = nltk.corpus.stopwords.words('english')

Step 4b: Tokenize Words & DIY Frequency Distribution

word_frequencies = {}
for word in nltk.word_tokenize(formatted_article_text):
    if word not in stopwords:
        if word not in word_frequencies.keys():
            word_frequencies[word] = 1
        else:
            word_frequencies[word] += 1

Step 4c: Calculate Weighted Frequency

max_frequency = max(word_frequencies.values())
for word in word_frequencies.keys():
    word_frequencies[word] = (word_frequencies[word]/max_frequency)

STEP 5: CALCULATE SENTENCE SCORES

## ILLUSTRATIVE EXAMPLE
## Nothing removed
for sent in sentence_list[:1]:
    for word in nltk.word_tokenize(sent.lower()):
        print(word)

bad
.

## ILLUSTRATIVE EXAMPLE
## Stopwords etc. removed
## We are ONLY assigning values/weights to the words in the sentences that are inside our freq dist!

for sent in sentence_list[:1]:
    for word in nltk.word_tokenize(sent.lower()):
        if word in word_frequencies.keys():
            print(word)

bad

sentence_scores = {}
for sent in sentence_list:
    for word in nltk.word_tokenize(sent.lower()):
        if word in word_frequencies.keys():
            if len(sent.split(' ')) < 200:
                if sent not in sentence_scores.keys():
                    sentence_scores[sent] = word_frequencies[word]
                else:
                    sentence_scores[sent] += word_frequencies[word]

sorted_sentences = sorted(sentence_scores.items(), key=lambda kv: kv[1], reverse=True)
sorted_sentences[:10]

[("it sure wasn't for their brutally bad singing voices ) , and the movie chronicles how their lives change for the worse as the pressures of fame get to them .everything from big egos , to booze and drugs to free flowing sex sends them on a downward spiral .there are a couple of other idiotic subplots thrown in for good measure , but the fame is the one that pretty much sums up this thing .>from a creative standpoint there is nothing redeeming here .other than the above-mentioned obsession with big knockers that russ meyer seemed to have .the dialogue is so incredibly bad that it literally is funny in parts .mr .",
  7.235294117647053),
 ('bad .', 0.47058823529411764)]

summary = [sent[0] for sent in sorted_sentences[:10]]
''.join(summary)

"it sure wasn't for their brutally bad singing voices ) , and the movie chronicles how their lives change for the worse as the pressures of fame get to them .everything from big egos , to booze and drugs to free flowing sex sends them on a downward spiral .there are a couple of other idiotic subplots thrown in for good measure , but the fame is the one that pretty much sums up this thing .>from a creative standpoint there is nothing redeeming here .other than the above-mentioned obsession with big knockers that russ meyer seemed to have .the dialogue is so incredibly bad that it literally is funny in parts .mr .bad ."

''.join(summary).strip()

'bad .'