2004
  1. Home
  2. A-Z Publications
  3. Computational Communication Research
  4. Volume 4, Issue 2
  5. Article
Volume 4, Issue 2
  • E-ISSN: 2665-9085
file format pdf download PDF

Abstract

Abstract

Topics and frames are at the heart of various theories in communication science and other social sciences, making their measurement of key interest to many scholars. The current study compares and contrasts two main deductive computational approaches to measure policy topics and frames: Dictionary (lexicon) based identification, and supervised machine learning. Additionally, we introduce domain-specific word embeddings to these classification tasks. Drawing on a manually coded dataset of Dutch news articles and parliamentary questions, our results indicate that supervised machine learning outperforms dictionary-based classification for both tasks. Furthermore, results show that word embeddings may boost performance at relatively low cost by introducing relevant and domain-specific semantic information to the classification model.

© Anne C. Kroon, Toni van der Meer & Rens Vliegenthart. This is an open access article distributed under the terms of the CC BY-NC 4.0 license. http://creativecommons.org/licenses/by/4.0
Loading

Article metrics loading...

/content/journals/10.5117/CCR2022.2.006.KROO
2022-10-01
2024-11-08
Loading full text...

Full text loading...

/deliver/fulltext/26659085/4/2/CCR2022.2.006.KROO.html?itemId=/content/journals/10.5117/CCR2022.2.006.KROO&mimeType=html&fmt=ahah

References

  1. Al-Azani, S., & El-Alfy, E. S. M. (2017). Using word embedding and ensemble learning for highly imbalanced data sentiment analysis in short Arabic text. Procedia Computer Science, 109, 359–366. https://doi.org/10.1016/j.procs.2017.05.365
     [Google Scholar]
  2. Albaugh, Q., Sevenans, J., Soroka, S., & Loewen, P. J. (2013). The automated coding of policy agendas: A dictionary-based approach. 6th Annual Comparative Agendas Conference, 1–22.
     [Google Scholar]
  3. Albaugh, Q., Soroka, S., Joly, J., Loewen, P., Sevenans, J., & Walgrave, S. (2014). Comparing and combining machine learning and dictionary-based approaches to topic coding. 7th Annual Comparative Agendas Project (CAP) Conference, March 2017, 1–18.
     [Google Scholar]
  4. Barberá, P., Boydstun, A. E., Linn, S., McMahon, R., & Nagler, J. (2021). Automated text classification of news articles: A practical guide. Political Analysis, 29(1), 19–42. https://doi.org/10.1017/pan.2020.8
     [Google Scholar]
  5. Bilbro, R., Ojeda, T., & Bengfort, B. (2018). Applied text analysis with Python. O’Reilly Media, Incorporated. https://books.google.nl/books?id=IrBqswEACAAJ
     [Google Scholar]
  6. Boukes, M., van de Velde, B., Araujo, T., & Vliegenthart, R. (2019). What’s the tone? Easy doesn’t do It: Analyzing performance and agreement between off-the-shelf sentiment analysis tools. Communication Methods and Measures, 00(00), 1–22. https://doi.org/10.1080/19312458.2019.1671966
     [Google Scholar]
  7. Boumans, J. (2017). Subsidizing the news?: Organizational press releases’ influence on news media’s agenda and content. Journalism Studies, 0(0), 1–19. https://doi.org/10.1080/1461670X.2017.1338154
     [Google Scholar]
  8. Boumans, J. W., & Trilling, D. (2016). Taking stock of the toolkit: An overview of relevant automated content analysis approaches and techniques for digital journalism scholars. Digital Journalism, 4(1), 8–23. https://doi.org/10.1080/21670811.2015.1096598
     [Google Scholar]
  9. Budak, C., Goel, S., & Rao, J. M. (2016). Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly, 80(Specialissue1), 250–271. https://doi.org/10.1093/poq/nfw007
     [Google Scholar]
  10. Burscher, Björn, Odijk, D., Vliegenthart, R., de Rijke, M., & de Vreese, C. H. (2014). Teaching the computer to code frames in news: Comparing two supervised machine learning approaches to frame analysis. Communication Methods and Measures, 8(3), 190–206. https://doi.org/10.1080/19312458.2014.937527
     [Google Scholar]
  11. Burscher, Bjorn, Vliegenthart, R., & De Vreese, C. H. (2015). Using supervised machine learning to code policy issues: Can classifiers generalize across contexts?Annals of the American Academy of Political and Social Science, 659(1), 122–131. https://doi.org/10.1177/0002716215569441
     [Google Scholar]
  12. Chan, C., Bajjalieh, J., Auvil, L., Wessler, H., Althaus, S., Welbers, K., van Atteveldt, W., & Jungblut, M. (2021). Four best practices for measuring news sentiment using ‘off-the-shelf’ dictionaries: a large-scale p-hacking experiment. Computational Communication Research, 3(1), 1–27. https://doi.org/10.5117/ccr2021.1.001.chan
     [Google Scholar]
  13. Colleoni, E., Rozza, A., & Arvidsson, A. (2014). Echo Chamber or Public Sphere? Predicting Political Orientation and Measuring Political Homophily in Twitter Using Big Data. Journal of Communication, 64(2), 317–332. https://doi.org/10.1111/jcom.12084
     [Google Scholar]
  14. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglue, K., & Kuksa, P. (2011). Natural Language Processing (almost) from scratch. Journal of Machine Learning Research, 12, 2493–2537. https://doi.org/10.1109/CIC.2017.00050
     [Google Scholar]
  15. Corrêa, E. A., Marinho, V. Q., Borges, L., & Santos, D. (2017). A multi-view ensemble for twitter sentiment analysis. Proceedings Ofthe 11th International Workshop on Semantic Evaluations (SemEval-2017), 611–615. https://business.twitter.com/en/basics.html
     [Google Scholar]
  16. Damstra, A., & Boukes, M. (2018). The economy, the news, and the public: A longitudinal study of the impact of economic news on economic evaluations and expectations. Communication Research, 009365021775097. https://doi.org/10.1177/0093650217750971
     [Google Scholar]
  17. Djerf-Pierre, M., & Shehata, A. (2017). Still an agenda setter: Traditional news media and public opinion during the transition from low to high choice media environments. Journal of Communication, 67(5), 733–757. https://doi.org/10.1111/jcom.12327
     [Google Scholar]
  18. Ferrero, J., Besacier, L., Agnes, F., & Schwab, D. (2017). Using word embedding for cross-language plagiarism detection. Proceedings Ofthe 15th Conference Ofthe European Chapter Ofthe Association for Computational Linguistics: Volume 2, Short Papers, 2, 415–421.
     [Google Scholar]
  19. Firth, J. R. (1935). The techniques of semantics. Transactions of Philological Society, 34(1), 36–77.
     [Google Scholar]
  20. Firth, J. R. (1957). Papers in linguistics, 1934-1951. Oxford University Press. https://books.google.nl/books?id=yxZZAAAAMAAJ
     [Google Scholar]
  21. Giatsoglou, M., Vozalis, M. G., Diamantaras, K., Vakali, A., Sarigiannidis, G., & Chatzisavvas, K. C. (2017). Sentiment analysis leveraging emotions and word embeddings. Expert Systems with Applications, 69, 214–224. https://doi.org/10.1016/j.eswa.2016.10.043
     [Google Scholar]
  22. Grimmer, J., & Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 267–297. https://doi.org/10.1093/pan/mps028
     [Google Scholar]
  23. Günther, E., & Quandt, T. (2016). Word counts and topic models: Automated text analysis methods for digital journalism research. Digital Journalism, 4(1), 75–88. https://doi.org/10.1080/21670811.2015.1093270
     [Google Scholar]
  24. Guo, L., & Vargo, C. (2015). The power of message networks: A big-data analysis of the network agenda setting model and issue ownership. Mass Communication and Society, 18(5), 557–576. https://doi.org/10.1080/15205436.2015.1045300
     [Google Scholar]
  25. Guo, L., Vargo, C. J., Pan, Z., Ding, W., & Ishwar, P. (2016). Big social data analytics in journalism and mass communication: Comparing dictionary-based text analysis and unsupervised topic modeling. Journalism and Mass Communication Quarterly, 93(2), 322–359. https://doi.org/10.1177/1077699016639231
     [Google Scholar]
  26. Hailong, Z., Wenyan, G., & Bo, J. (2014). Machine learning and lexicon based methods for sentiment classification: A survey. Proceedings – 11th Web Information System and Application Conference, WISA 2014, 262–265. https://doi.org/10.1109/WISA.2014.55
     [Google Scholar]
  27. Hertog, J. K., & McLeod, D. M. (2001). A multiperspectival approach to framing analysis: A field guide. In Framing public life (pp. 157–178). Routledge.
     [Google Scholar]
  28. King, G., Lam, P., & Roberts, M. E. (2017). Computer-Assisted Keyword and Document Set Discovery from Unstructured Text. American Journal of Political Science, 61(4), 971–988. https://doi.org/10.1111/ajps.12291
     [Google Scholar]
  29. King, G., Schneer, B., & White, A. (2017). How the news media activate public expression and influence national agendas. Science, 358(6364), 776–780.
     [Google Scholar]
  30. Kroon, A. C., & van der Meer, T. G. L. A. (2018). Who Takes the Lead? Investigating the Reciprocal Relationship Between Organizational and News Agendas. Communication Research. https://doi.org/10.1177/0093650217751733
     [Google Scholar]
  31. Kübler, R. V., Wieringa, J. E., & Pauwels, K. H. (2017). Machine learning and big data. In P. K.LeeflangP., WieringaJ., Bijmolt T. (Ed.), Advanced Methods for Modeling Markets. International Series in Quantitative Marketing. Springer.
     [Google Scholar]
  32. Le, Q.,& Mikolov, T. (2014). Distributed representations of sentences and documents. Proceedings of the 31st International Conference on International Conference on Machine Learning – Volume 32, 1188--1196. http://dl.acm.org/citation.cfm?id=3044805.3045025
     [Google Scholar]
  33. Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. Journal of Finance, 66(1), 35–65. https://doi.org/10.1111/j.1540-6261.2010.01625.x
     [Google Scholar]
  34. Maher, M. (2001). Framing: An Emerging Paradigm or A Phase of Agenda Setting?Framing Public Life: Perspectives on Media and Our Understanding of the Social World, 1972, 83–94.
     [Google Scholar]
  35. Matthes, J., & Kohring, M. (2008). The content analysis of media frames: Toward improving reliability and validity. Journal of Communication, 58(2), 258–279. https://doi.org/10.1111/j.1460-2466.2008.00384.x
     [Google Scholar]
  36. Mikolov, T., Corrado, G., Chen, K., & Dean, J. (2013). Efficient estimation of word representations in vector space. ArXiv, 1–12.
     [Google Scholar]
  37. Neuman, W. R., Guggenheim, L., Jang, S. M., & Bae, S. Y. (2014). The dynamics of public attention: Agenda-setting theory meets big data. Journal of Communication, 64(2), 193–214. https://doi.org/10.1111/jcom.12088
     [Google Scholar]
  38. Opperhuizen, A. E., Schouten, K., & Klijn, E. H. (2019). Framing a conflict! How media report on earthquake risks caused by gas drilling: A longitudinal analysis using machine learning techniques of media reporting on gas drilling from 1990 to 2015. Journalism Studies, 20(5), 714–734. https://doi.org/10.1080/1461670X.2017.1418672
     [Google Scholar]
  39. Ribeiro, F., Araújo, M., Gonçalves, P., Gonçalves, M., & Benevenuto, F. (2015). Senti-Bench – A benchmark comparison of state-of-the-practice sentiment analysis methods. Arxiv, 9(4), 1–32. http://arxiv.org/abs/1512.01818
     [Google Scholar]
  40. Rudkowsky, E., Haselmayer, M., Wastian, M., Jenny, M., Emrich, Š., Sedlmair, M. (2018). More than bags of words: Sentiment analysis with word embeddings. Communication Methods and Measures, 12(2–3), 140–157. https://doi.org/10.1080/19312458.2018.1455817
     [Google Scholar]
  41. Ruigrok, N., & Atteveldt, W. Van. (2007). Global angling with a local angle: How U.S., British, and Dutch newspapers frame global and local terrorist attacks. Press/Politics, 12(1), 68–90. https://doi.org/10.1177/1081180X06297436
     [Google Scholar]
  42. Semetko, H. A., & Valkenburg, P. M. (2000). Framing European politics: A content analysis of press and television news. Journal of Communication, 50(2), 93–109. https://doi.org/10.1111/j.1460-2466.2000.tb02843.x
     [Google Scholar]
  43. Simon, A. F. (2001). A unified method for analyzing media framing. Communication in US Elections: New Agendas, 75–89.
     [Google Scholar]
  44. Stoll, A., Ziegele, M., & Quiring, O. (2020). Detecting Impoliteness and Incivility in Online Discussions. Computational Communication Research, 2(1), 109–134. https://doi.org/10.5117/ccr2020.1.005.kath
     [Google Scholar]
  45. Su, L. Y. F., Xenos, M. A., Rose, K. M., Wirz, C., Scheufele, D. A., & Brossard, D. (2018). Uncivil and personal? Comparing patterns of incivility in comments on the Facebook pages of news outlets. New Media and Society, 20(10), 3678–3699. https://doi.org/10.1177/1461444818757205
     [Google Scholar]
  46. Taboada, M., Brooke, J., & Voll, K. (2011). Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2).
     [Google Scholar]
  47. Tamara, K., & Milićević, N. (2018). Comparing sentiment analysis and document representation methods of Amazon reviews. SISY 2018 – IEEE 16th International Symposium on Intelligent Systems and Informatics, Proceedings, 283–288. https://doi.org/10.1109/SISY.2018.8524814
     [Google Scholar]
  48. Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010). Sentiment in short strength detection informal text. Journal of the American Society for Information Science and Technology, 6(12), 2544–2558. https://doi.org/10.1002/asi.21416
     [Google Scholar]
  49. Tulkens, S., Hilte, L., Lodewyckx, E., Verhoeven, B., & Daelemans, W. (2016). A dictionary-based approach to racism detection in Dutch social media. https://doi.org/None
     [Google Scholar]
  50. Van Aelst, P., & Vliegenthart, R. (2014). Studying the Tango: An analysis of parliamentary questions and press coverage in the Netherlands. Journalism Studies, 15(4), 392–410. https://doi.org/10.1080/1461670X.2013.831228
     [Google Scholar]
  51. Van der Pas, D. (2013). Making hay while the sun shines: Do parties only respond to media attention when the framing is right?The International Journal of Press/ Politics, 19(1), 42–65. https://doi.org/10.1177/1940161213508207
     [Google Scholar]
  52. Vargo, C. J., Guo, L., Mccombs, M., & Shaw, D. L. (2014). Network issue agendas on Twitter during the 2012 U.S. presidential election. Journal of Communication, 64(2), 296–316. https://doi.org/10.1111/jcom.12089
     [Google Scholar]
  53. Vliegenthart, R., Walgrave, S., Baumgartner, F. R., Bevan, S., Breunig, C., Brouard, S., Bonafont, L. C., Grossman, E., Jennings, W., Mortensen, P. B., Palau, A. M., Sciarini, P., & Tresch, A. (2016). Do the media set the parliamentary agenda? A comparative study in seven countries. European Journal of Political Research, 55(2), 283–301. https://doi.org/10.1111/1475-6765.12134
     [Google Scholar]
  54. Welbers, K., & Opgenhaffen, M. (2019). Presenting News on Social Media: Media logic in the communication style of newspapers on Facebook. Digital Journalism, 7(1), 45–62. https://doi.org/10.1080/21670811.2018.1493939
     [Google Scholar]
  55. Yuan, X., Schuchard, R. J., & Crooks, A. T. (2019). Examining Emergent Communities and Social Bots Within the Polarized Online Vaccination Debate in Twitter. Social Media + Society, 5(3), 205630511986546. https://doi.org/10.1177/2056305119865465
     [Google Scholar]
/content/journals/10.5117/CCR2022.2.006.KROO
Loading
Beyond Counting Words
CCR 4, 528 (2022); https://doi.org/10.5117/CCR2022.2.006.KROO
/content/journals/10.5117/CCR2022.2.006.KROO
/content/journals/10.5117/CCR2022.2.006.KROO
Loading

Data & Media loading...

Keyword(s): automated text analysis; dictionaries; frames; policy topics; supervised machine learning; word embeddings

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error