Team solutions

CASE Ontotext, Team CENTROIDA

This paper presents a DNN-based approach to learn entities relations from distant-labeled free text. The proposed approach presents task-specific data cleaning, which despite effective in removing textual noise is preserving semantics necessary for the training process. The cleaned-up dataset is then used to build a number of bLSTM attention-based DNN models, hyper-tuned using recall as an optimization objective. The resulting models are then joined into an ensemble that deliver our best result

2
votes

8 thoughts on “CASE Ontotext, Team CENTROIDA

    1. 0
      votes

      We could, but.. we’re honestly quite tired, so we’ll skip this now and leave it for later on. It’s not relevant for the overall task anyways:

      From the Ontotext case – ” … The teams will only need to identify if there is a relation of type _is parent of_ . … “.

      We’ve done that – is_parent column (like in the training data) is present 🙂

  1. 1
    votes

    Good understanding of the business problem and a good choice for an algorithm that can tackle the task. I would have liked to learn a bit more about the nitty-gritty of the solution from the article- what was the structure of the best performing model, how was data split, what data were recall and accuracy calculated on.

    I’m looking forward to see how well your approach did on the test data as the results you’ve gotten are quite promising.

  2. 1
    votes

    Nice work! You have found a very relevant paper, by a world-top research group, and it was further extended, based on ideas from another paper, and using improvements on the network architecture, and based on exploration of the values of the hyper-parameters. The model uses deep learning and state-of-the-art tools and techniques.

    How were the company names normalized exactly?

    Do you do anything special to handle the asymmetricity of the relation?

    The accuracy is very high, but what is the baseline? Also, what is F1?
    Any results on cross-validation based on the training dataset for different choices of the hyperparameters of the network architecture?

    Any thought what can be done next to further improve the model?

    1. 0
      votes

      Thanks for the feedback, Preslav. Much appreciated.

      Normalization – replace the column1/column2 ocurrences in the snippet with . All our extra vocabulary has this syntax

      We do nothing special for the asymmetricity other than the used model architectures (blstms should be quite contributing to tackling this subtask)

      We’ll post the F1 and other data in a separate blog post.. Didn’t have much time to crunch these numbers as well

      As for improvement – yup. Based on gut-feeling confidence, we think can squeeze out at least 2-3% out of this, mostly via larger hyperparam search

      Cheers,
      A

  3. 1
    votes

    Great work! I agree with the mentors above 🙂 You built a great model and achieved a very high score! Great productivity for a group of 3 people. I would also like to see more graphics, scores, etc.
    Here are some notes I made reading your article:
    – the dataset is not that biased, negative examples are not orders of magnitude larger than positive
    – normalizing the company names is a very good idea
    – stopwords may bring value in some cases, always test and verify if removing them actually helps
    – A_team noted that in many examples the text doesn’t hold enough information about the relation between the two companies (producing erroneous examples). Did you observe that and did you try to handle it?
    – If the results are on the test set, great results! Very good application of neural networks.

Leave a Reply