{"id":240,"date":"2021-07-14T16:58:34","date_gmt":"2021-07-14T20:58:34","guid":{"rendered":"https:\/\/arriablog22.wpengine.com\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/"},"modified":"2022-03-24T15:26:11","modified_gmt":"2022-03-24T19:26:11","slug":"finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives","status":"publish","type":"post","link":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/","title":{"rendered":"Finding creative solutions to detect mistakes in neural-NLG narratives"},"content":{"rendered":"<p>There is a lot of excitement in the NLG world about using neural (deep-learning) techniques to build NLG systems. However, one big problem with current neural NLG systems is that the narratives they generate can contain a lot of factual errors, especially if the narratives are longer than 10-20 words.<\/p>\n<p> <!--more--> <\/p>\n<p>I am working with an Aberdeen University PhD student, Craig Thomson, on trying to better understand the number and type of mistakes made, and how they can be detected. After all, if an NLG system can detect when it has produced a flawed narrative, it could try again, or ask a person (\u201chuman in the loop\u201d) to fix the narrative.<\/p>\n<p>Craig and I have our own ideas about this, but we\u2019ve been keen to get ideas from other people as well, so we decided to run a \u201cshared task\u201d on detecting mistakes. Shared tasks are basically contests, where the organizers (i.e., Craig and I) propose a well-defined problem, and ask participants to submit solutions (e.g., algorithms). Anyone can participate, so it\u2019s a great way to get ideas from people with very different backgrounds!<\/p>\n<p><a href=\"https:\/\/www.aclweb.org\/anthology\/2020.inlg-1.28\/\" rel=\"noopener\" target=\"_blank\">We described the shared task in a paper<\/a>, and full details are <a href=\"https:\/\/github.com\/ehudreiter\/accuracySharedTask\" rel=\"noopener\" target=\"_blank\">available at our GitHub site<\/a> (warning: the site is pretty technical). Basically, we asked participants to develop algorithms and protocols that can find mistakes in summaries of basketball games produced by neural NLG systems. We manually analyzed the mistakes in 60 such summaries, and the participants used this \u201ctraining data\u201d to build and tune their algorithms and protocols, which they sent to us on 15 June. We\u2019re now in the process of checking how well these approaches work on a \u201ctest set\u201d of 30 additional summaries of basketball games. Results will be announced at the <a href=\"https:\/\/inlg2021.github.io\/\" rel=\"noopener\" target=\"_blank\">INLG conference in Aberdeen in September<\/a>.<\/p>\n<p>It\u2019s exciting and I\u2019m looking forward to announcing the results! We have participants from three continents who are exploring very different approaches. It\u2019s fascinating to see which approaches work well and which do not; and if some of the approaches are successful, this will be a huge help in building useful neural NLG systems!<\/p>\n<p>I\u2019d also encourage anyone who is interested in accuracy of neural NLG to look at the GitHub site for our shared task, especially <a href=\"https:\/\/github.com\/ehudreiter\/accuracySharedTask\/blob\/main\/gsml.csv\" rel=\"noopener\" target=\"_blank\">gsml.csv<\/a> (which is the manually produced list of mistakes in the 60 original narratives). I\u2019ve seen a lot of discussion of accuracy problems, which is vague and not based on real data. Our <a href=\"https:\/\/github.com\/ehudreiter\/accuracySharedTask\/blob\/main\/gsml.csv\" rel=\"noopener\" target=\"_blank\">gsml.csv<\/a> file lists 1,214 mistakes in 60 narratives, and I think it\u2019s very helpful in giving a concrete understanding of the number and type of errors produced by neural NLG systems in this kind of task.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>There is a lot of excitement in the NLG world&#8230;<\/p>\n","protected":false},"author":10,"featured_media":1116,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[34,36,35,29],"class_list":["post-240","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","tag-ai","tag-neural","tag-nlg-applications","tag-nlg-blog"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Finding creative solutions to detect mistakes in neural-NLG narratives - NLG Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Finding creative solutions to detect mistakes in neural-NLG narratives - NLG Blog\" \/>\n<meta property=\"og:description\" content=\"There is a lot of excitement in the NLG world...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/\" \/>\n<meta property=\"og:site_name\" content=\"NLG Blog\" \/>\n<meta property=\"article:published_time\" content=\"2021-07-14T20:58:34+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-03-24T19:26:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/BL133-EhudFinding2-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"790\" \/>\n\t<meta property=\"og:image:height\" content=\"334\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Ehud Reiter\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ehud Reiter\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/\"},\"author\":{\"name\":\"Ehud Reiter\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#\\\/schema\\\/person\\\/31224a0ca3829f43a13781b3d7afd7e0\"},\"headline\":\"Finding creative solutions to detect mistakes in neural-NLG narratives\",\"datePublished\":\"2021-07-14T20:58:34+00:00\",\"dateModified\":\"2022-03-24T19:26:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/\"},\"wordCount\":459,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/3\\\/2022\\\/03\\\/BL133-EhudFinding2-1.jpg\",\"keywords\":[\"AI\",\"Neural\",\"NLG applications\",\"NLG Blog\"],\"articleSection\":[\"NLG Blog\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/\",\"url\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/\",\"name\":\"Finding creative solutions to detect mistakes in neural-NLG narratives - NLG Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/3\\\/2022\\\/03\\\/BL133-EhudFinding2-1.jpg\",\"datePublished\":\"2021-07-14T20:58:34+00:00\",\"dateModified\":\"2022-03-24T19:26:11+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/3\\\/2022\\\/03\\\/BL133-EhudFinding2-1.jpg\",\"contentUrl\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/3\\\/2022\\\/03\\\/BL133-EhudFinding2-1.jpg\",\"width\":790,\"height\":334,\"caption\":\"BL133-EhudFinding2\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Finding creative solutions to detect mistakes in neural-NLG narratives\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/\",\"name\":\"NLG Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#organization\",\"name\":\"NLG Blog\",\"url\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/3\\\/2022\\\/03\\\/arria_logo_125x30.png\",\"contentUrl\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/3\\\/2022\\\/03\\\/arria_logo_125x30.png\",\"width\":125,\"height\":30,\"caption\":\"NLG Blog\"},\"image\":{\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/#\\\/schema\\\/person\\\/31224a0ca3829f43a13781b3d7afd7e0\",\"name\":\"Ehud Reiter\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/b2b51dd76543bef69265c1b1b8d995a0132ea071f50988250c00fdea10b15bf9?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/b2b51dd76543bef69265c1b1b8d995a0132ea071f50988250c00fdea10b15bf9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/b2b51dd76543bef69265c1b1b8d995a0132ea071f50988250c00fdea10b15bf9?s=96&d=mm&r=g\",\"caption\":\"Ehud Reiter\"},\"url\":\"https:\\\/\\\/www.arria.com\\\/blog\\\/author\\\/ehud-reiter\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Finding creative solutions to detect mistakes in neural-NLG narratives - NLG Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/","og_locale":"en_US","og_type":"article","og_title":"Finding creative solutions to detect mistakes in neural-NLG narratives - NLG Blog","og_description":"There is a lot of excitement in the NLG world...","og_url":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/","og_site_name":"NLG Blog","article_published_time":"2021-07-14T20:58:34+00:00","article_modified_time":"2022-03-24T19:26:11+00:00","og_image":[{"width":790,"height":334,"url":"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/BL133-EhudFinding2-1.jpg","type":"image\/jpeg"}],"author":"Ehud Reiter","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ehud Reiter","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#article","isPartOf":{"@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/"},"author":{"name":"Ehud Reiter","@id":"https:\/\/www.arria.com\/blog\/#\/schema\/person\/31224a0ca3829f43a13781b3d7afd7e0"},"headline":"Finding creative solutions to detect mistakes in neural-NLG narratives","datePublished":"2021-07-14T20:58:34+00:00","dateModified":"2022-03-24T19:26:11+00:00","mainEntityOfPage":{"@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/"},"wordCount":459,"commentCount":0,"publisher":{"@id":"https:\/\/www.arria.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#primaryimage"},"thumbnailUrl":"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/BL133-EhudFinding2-1.jpg","keywords":["AI","Neural","NLG applications","NLG Blog"],"articleSection":["NLG Blog"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/","url":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/","name":"Finding creative solutions to detect mistakes in neural-NLG narratives - NLG Blog","isPartOf":{"@id":"https:\/\/www.arria.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#primaryimage"},"image":{"@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#primaryimage"},"thumbnailUrl":"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/BL133-EhudFinding2-1.jpg","datePublished":"2021-07-14T20:58:34+00:00","dateModified":"2022-03-24T19:26:11+00:00","breadcrumb":{"@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#primaryimage","url":"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/BL133-EhudFinding2-1.jpg","contentUrl":"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/BL133-EhudFinding2-1.jpg","width":790,"height":334,"caption":"BL133-EhudFinding2"},{"@type":"BreadcrumbList","@id":"https:\/\/www.arria.com\/blog\/finding-creative-solutions-to-detect-mistakes-in-neural-nlg-narratives\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.arria.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Finding creative solutions to detect mistakes in neural-NLG narratives"}]},{"@type":"WebSite","@id":"https:\/\/www.arria.com\/blog\/#website","url":"https:\/\/www.arria.com\/blog\/","name":"NLG Blog","description":"","publisher":{"@id":"https:\/\/www.arria.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.arria.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.arria.com\/blog\/#organization","name":"NLG Blog","url":"https:\/\/www.arria.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.arria.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/arria_logo_125x30.png","contentUrl":"https:\/\/www.arria.com\/blog\/wp-content\/uploads\/sites\/3\/2022\/03\/arria_logo_125x30.png","width":125,"height":30,"caption":"NLG Blog"},"image":{"@id":"https:\/\/www.arria.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.arria.com\/blog\/#\/schema\/person\/31224a0ca3829f43a13781b3d7afd7e0","name":"Ehud Reiter","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/b2b51dd76543bef69265c1b1b8d995a0132ea071f50988250c00fdea10b15bf9?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/b2b51dd76543bef69265c1b1b8d995a0132ea071f50988250c00fdea10b15bf9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b2b51dd76543bef69265c1b1b8d995a0132ea071f50988250c00fdea10b15bf9?s=96&d=mm&r=g","caption":"Ehud Reiter"},"url":"https:\/\/www.arria.com\/blog\/author\/ehud-reiter\/"}]}},"_links":{"self":[{"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/posts\/240","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/comments?post=240"}],"version-history":[{"count":0,"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/posts\/240\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/media\/1116"}],"wp:attachment":[{"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/media?parent=240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/categories?post=240"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.arria.com\/blog\/wp-json\/wp\/v2\/tags?post=240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}