Hack the News Datathon 2019

Posted balavignesh95, kabilarasan, balavignesh95, prafuldev, yashwanth, saunak13 Mentors: drsubhabahapal, preslav, alberto, giovanni, laura, vsenderovLeave a commentPosted in Classification systemsClassification systems

Datathon – HackNews – Solution – Insight Hunters

In order to do the following we have to undergo the process of text cleaning, understanding the text. We had to find a way in order to split the data and form a data frame which consists of the following columns.News_TextNews_NumberNews_TypeThe data has lots of fillers which had to be removed and some rows where news_numbers and type were missing. In order to clean the data we had to remove the fillers using the NLTK stop words filtration. Later on we tokenized the data using the word_tokenizer from the nltk package.The next important step was to lemmatize/stem the data to remove the tense from the words and normalize the words. Even though it was a time consumption process the results were promising.XGBoost has capability to handle the imbalanced dataset by using the parameter Scale_Pos_weight. we can do threshold probability throttling to increase the sensitivity by sacrificing the reasonable specificity.Evaluation:- This process is kind of tricky for the train data set provided, as the data was highly imbalanced, the dependent feature/variable had imbalanced classes

Posted omki, abhinandanarya, apoorvakesarwani, shivangikaul Mentors: drsubhabahapal, preslav, laura, vsenderov, giovanni, albertoLeave a commentPosted in Datathons SolutionsDatathons Solutions

Datathon-HackNews-Solutions-Minions

We are back to participate in another Datathon hosted by Data Science Society. This time the theme is Text Analytics.
We will not be able to completely devote ourselves to the cause this time because of the exams which start in next week. But We’ll try to keep the article as simple and well detailed as possible so that it will be helpful for any new Data Science Enthusiast seeking for little helps. So Lets roll.

Posted ananya, ananya, gayathri, apoo, krishna, sainadhamul1 Mentors: drsubhabahapal, senthil, preslav, laura, vsenderovLeave a commentPosted in Datathons SolutionsDatathons Solutions

Datathon – HackNews – Solution – TEXT_MINERS

Propaganda is a form of communication that is aimed at influencing the attitude of a community toward some cause or position. It often presents facts selectively to encourage a particular synthesis. The disinformation damages the reputation of respectable news outlets, organisations and very bad for business indeed. The objective of the Hackathon is to be able to detect the Propaganda and Non-propaganda news as well as to develop a model that can help with the venture. The other objectives of this work includes detecting phrases which are propagandist and also finding out the type of propaganda it is. The algorithms that we will be taking help from are Passive Aggressive, Multiple Layer Perceptron Network, Logistic Regression, AdaBoost, Decision Tree, Random Forest, KNN, SVM and Naive Bayes to detect the potentially propagandistic and non-propagandistic sentences in a news article. For the evaluation, we are calculating F1 Score to measure the class imbalance in the testing dataset. We have used the best model for detecting propagandist and non-propagandist articles, phrases and also type of propaganda.

Posted vedh, ganeshnethi, frankenstein, rithvik, pmohan, vedh Mentors: drsubhabahapal, preslav, alberto, giovanni, laura, vsenderov, senthilLeave a commentPosted in Datathons SolutionsDatathons Solutions

Datathon-HackNews-Solution-indianhunters

News is the lifeline of the human society , it underlines all the important events and influences public opinion like no other tool , but with the recent advent of electronic media and the sheer amount of new being churned out and the current political climate it’s hard to figure out what’s genuine news and what’s propaganda , this is where intelligent systems which can classify news articles , text fragments as propagandistic or non-propagandistic comes into play , this Datathon is focussed on developing such a system using various algorithms and methods to predict such a scenario the levels of challenges are:

A System that is able to classify a news article whether it is propaganda or not.
A System that is able to classify whether a sentence in a article is propaganda or not.
A System that is intelligently able to classify the propaganda technique used in the new piece.

Posted sanjeevbiswas05, dhiru29, sunny, gourang, sanjeevbiswas05 Mentors: senthil, drsubhabahapal, preslav, laura, vsenderovLeave a commentPosted in Datathons SolutionsDatathons Solutions

Datathon – HackNews – Solution – Path Finders

In recent years, deceptive content such as fake news and fake reviews, also known as opinion spams, have increasingly become a dangerous prospect for online users. Fake reviews have affected consumers and stores alike. Furthermore, the problem of fake news has gained attention in 2016, especially in the aftermath of the last U.S. presidential elections. Fake reviews and fake news are a closely related phenomenon as both consist of writing and spreading false information or beliefs. The opinion spam problem was formulated for the first time a few years ago, but it has quickly become a growing research area due to the abundance of user-generated content. It is now easy for anyone to either write fake reviews or write fake news on the web. The biggest challenge is the lack of an efficient way to tell the difference between a real review and a fake one; even humans are often unable to tell the difference. We are implementing  7 machine learning classification techniques here.

Posted subbu, shekharharshvardhan, ajaykalbavi123, jk8460, shivam Mentors: drsubhabahapal, senthil, preslav, laura, vsenderovLeave a commentPosted in Datathons SolutionsDatathons Solutions

Datathon – HackNews – Solution – Dream_Invaders

Due to the extreme divergence of social discussions happening in the political space, rumours and fake news becoming inferno which is difficult for anyone who reads to differentiate it from the truth.
What we are going to achieve?
To detect the propagandas at article level, sentence level and recognizing its type.
Using supervised machine learning technique, model shall be created to identify and flag the false news propaganda.

Posted ahmed, shehel, blackhoodie Mentors: preslav, laura, vsenderov, giovanni, alberto5 CommentsPosted in Datathons SolutionsDatathons Solutions

Datathon – HackNews – Solution – DataExploiters

This article describes our submission for the Hack the News Datathon 2019 which focuses on Task 2, Propaganda sentence classification. It outlines our exploratory data analysis, methodology and future work. Our work revolves around the BERT model as we believe it offers an excellent language model that’s also good at attending to context which is an important aspect of propaganda detection.

Posted stephen1 CommentPosted in ProbabilitiesProbabilities

Stochastic Processes and Applications

This notebook is a basic introduction into Stochastic Processes. It is meant for the general reader that is not very math savvy, like the course participants in the Math Concepts for Developers in SoftUni.
There is a basic definition. Some examples of the most popular types of processes like Random Walk, Brownian Motion or Weiner Process, Poisson Process and Markov chains have been given. Their basic characteristics and examples for some possible applications are stated. For all the examples there are simulations in Python, some are visualized.
The following packages have been used:

nympy
matplotlib.pyplot
random
scipy.stats
itertools
matplotlib.patches

Posted vpekar, wariodoor Mentors: preslav, vsenderov, laura, alberto, giovanni7 CommentsPosted in Datathons SolutionsDatathons Solutions

Datathon – HackNews – Solution – Leopards

This is a Leopards team’s submission for the Propaganda Detection datathon. Key findings: the best performing classifier is logistic regression, operating on Word2vec representations of the sentences plus several designed features like the proportion of sentiment-bearing words in the sentence.

Posted codenstein, saiaditya-g, kbarun, prasad, sreekar Mentors: senthil, drsubhabahapal, preslav, laura, vsenderov6 CommentsPosted in Datathons SolutionsDatathons Solutions

Datathon – HackNews – Solution – data_monks

The word propaganda is defined as designating any attempt to influence the opinions or actions of others to some predetermined end by appealing to their emotions or prejudices or by distorting the facts. We are fooled by propaganda chiefly because they appeal to our emotions rather than to our reason. They make us believe and do something we would not believe or do. And since it appeal more to our emotions; we often don’t recognize it when we see it.
The current political landscape is shaped by extreme polarization of opinions and by the proliferation of fake news.
Studies and surveys has found that rumour’s and fake news tend to spread six times faster than truthful information. This situation both damages the reputation of respectable news outlets and it also undermines the very foundations of democracy, which needs free and reliable press to thrive. Therefore, it is in the interest of the public as well as of the news organizations to be able to detect and fight disinformation in all its forms.
Here, we are trying to create a tool that can help identify propagandistic articles with the help of Predictive Analytics.
The main objectives are:
(i) to flag the article as a whole
(ii) to detect the potentially propagandistic sentences in a news article
(iii) to identify the exact type and span of use of propagandistic techniques