{"id":414,"date":"2018-07-08T16:21:25","date_gmt":"2018-07-08T16:21:25","guid":{"rendered":"http:\/\/ajhassoc.com\/?p=414"},"modified":"2018-07-08T16:21:25","modified_gmt":"2018-07-08T16:21:25","slug":"statistical-testing-is-essential-to-transportation-research","status":"publish","type":"post","link":"https:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/","title":{"rendered":"Statistical Testing is Essential to Transportation Research"},"content":{"rendered":"
\n

I have lately refereed journal submissions where authors have totally evaded the concept of a statistical test while trying to demonstrate the importance of their research.\u00a0 I will give somewhat altered (to protect the identities of the authors and avoid lawsuits) examples later.\u00a0 Mainly, each of these authors has developed an innovation, compared the innovation to a straw-man technique (or two), and concluded that their innovation is good.\u00a0 However, in many cases the slight differences in performance would likely fail a statistical test had a statistical test been performed.<\/p>\n

Witness the \u201cReproducibility Project: Psychology\u201d. \u00a0When experiments cannot be replicated, the cause might be bad statistics, but the cause could also involve investigator bias.\u00a0 In transportation we see much research with no statistical testing at all.\u00a0 I suspect that a lot of what has been published in our journals is unusable at best and misleading at worst.<\/p>\n

When we do statistical estimation with canned software, the more important statistical tests are automatically generated.\u00a0 They are unavoidable and referees know to ask for them.\u00a0 However, when developing new methods with custom code, those lovely statistical tests are often absent.\u00a0 The writing of a custom statistical test can be difficult.<\/p>\n

What does a statistical test do?\u00a0 If we determine experimentally that A is better than B, then a significant statistical test will tell us that if we were to repeat the experiment, thereby obtaining entirely new data, then it is highly likely that A will still be better than B.\u00a0 This also applies to data we collect through surveys or data that are obtained from traffic detectors.\u00a0 If an innovation is best within a single trial, it does not mean it will be best on the next trial, unless we have statistics to lend confidence to that assertion.<\/p>\n

There are rare cases where we might be willing to accept the claim of an innovation without a demonstration of statistical significance.\u00a0 Maybe an author has done an outstanding job of developing a new theory, or maybe the test data set is not entirely appropriate or available.\u00a0 In these cases the referees must make judgement calls as to whether the potential value of the innovation is worth the risk of a wrong conclusion.\u00a0 In these situations it is absolutely essential that the author provide the appropriate caveats.<\/p>\n

I am dismissing the possibility of out-right deception.\u00a0 I personally know of only one instance of deception happening, and this instance came from a colleague of mine rather than from a random author. \u00a0My unscientific opinion is that most wrong conclusions stem from investigator bias fueled by wishful thinking. \u00a0An author accrues far more benefits from getting a shaky paper into print than he or she risks losing by publishing bad results.<\/p>\n

Case Study A.\u00a0 An author proposes a method for forecasting a composite traffic variable one day ahead using only information about that traffic variable from the day before.\u00a0 He finds that he needs just three equations.\u00a0 One equation predicts Tuesday through Friday and Sunday.\u00a0 Another equation predicts Saturday.\u00a0 And the last equation predicts Monday.\u00a0 He ignores seasonality, but does not attempt to do anything with holidays.\u00a0 He calibrates his equations on a year\u2019s worth of data, but tests the results on a 6-day hold out sample.\u00a0 He compares his method to simple averages for each day of week, but develops a different average for each season.\u00a0 The author reports that the day-ahead method has a standard error of 11.1% and the averaging method has a worse standard error of 12.3%.\u00a0 Anyone who has ever worked with the F-test knows immediately that a difference of 1.2% is unlikely to be significant with only 6 samples.\u00a0 The author still claims his innovation is a success, but the author never did a statistical test to prove it.<\/p>\n

Case Study B.\u00a0 An author develops an artificial intelligence technique to do a short-term (less than 1 hour) forecast of a traffic speed.\u00a0 His technique is a modification of an existing technique.\u00a0 The author compares his innovation to the existing technique and a well-know time series method.\u00a0 Inexplicably, the author omits a rather important variable, included in the AI techniques, from the time-series method.\u00a0 The author uses a month of data across 10 detectors to train\/calibrate the techniques, then tests them on 5 full days of traffic data, divided into 15-minute intervals.\u00a0 This gives him a lot of samples in his test data set.\u00a0 His technique, of course does best, with an standard error of 4.80 mph against a standard error of 4.88 mph for the existing AI technique and 6.11 mph for the time-series technique.\u00a0 A statistical test would be difficult to perform because his data is seriously heteroskedastic and serially correlated, so the assumptions of elementary statistical tests are violated.\u00a0 It looks to me like the difference between the two AI techniques are too small to matter.\u00a0 And while the innovation outperforms the time-series method, the comparison is flawed because of an unlevel playing field.<\/p>\n

Note:\u00a0 These case studies are lightly fictionalized from actual papers submitted to a reputable academic journal.\u00a0 I also simplified some of the statistical issues to speed up this discussion, but I think I got the essence of the situations correct.\u00a0 Most referees thought these papers were worthy of publication.<\/p>\n

Believe me, the omission of a quality statistical test is not rare and many referees are quite lax in holding the authors to account.<\/p>\n

Neil Turok recently said, “All of the theoretical work that’s been done since the 1970s has not produced a single successful prediction.\u201d\u00a0 He is speaking of physics, not traffic or travel forecasting, but we are in danger of falling into the same predicament.<\/p>\n

We need to avoid granting free passes to authors who omit careful validation of their work or do so too casually.<\/p>\n

As always, I am interested in hearing your opinion.<\/p>\n

Alan Horowitz, Whitefish Bay, July 8, 2018<\/em><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

I have lately refereed journal submissions where authors have totally evaded the concept of a statistical test while trying to demonstrate the importance of their research.\u00a0 I will give somewhat altered (to protect the identities of the authors and avoid lawsuits) examples later.\u00a0 Mainly, each of these authors has developed an innovation, compared the innovation …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"enabled":false},"version":2}},"categories":[11,7,10],"tags":[],"class_list":["post-414","post","type-post","status-publish","format-standard","category-data","category-research-philosophy","category-travel-modeling","anons"],"yoast_head":"\nStatistical Testing is Essential to Transportation Research - QRS II Page<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Statistical Testing is Essential to Transportation Research - QRS II Page\" \/>\n<meta property=\"og:description\" content=\"I have lately refereed journal submissions where authors have totally evaded the concept of a statistical test while trying to demonstrate the importance of their research.\u00a0 I will give somewhat altered (to protect the identities of the authors and avoid lawsuits) examples later.\u00a0 Mainly, each of these authors has developed an innovation, compared the innovation ...\" \/>\n<meta property=\"og:url\" content=\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/\" \/>\n<meta property=\"og:site_name\" content=\"QRS II Page\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-08T16:21:25+00:00\" \/>\n<meta name=\"author\" content=\"horowitz\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"horowitz\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#article\",\"isPartOf\":{\"@id\":\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/\"},\"author\":{\"name\":\"horowitz\",\"@id\":\"https:\/\/ajhassoc.com\/#\/schema\/person\/b982087d07e1397f2767543367fb6070\"},\"headline\":\"Statistical Testing is Essential to Transportation Research\",\"datePublished\":\"2018-07-08T16:21:25+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/\"},\"wordCount\":976,\"commentCount\":0,\"articleSection\":[\"Data\",\"Research Philosophy\",\"Travel Modeling\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/\",\"url\":\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/\",\"name\":\"Statistical Testing is Essential to Transportation Research - QRS II Page\",\"isPartOf\":{\"@id\":\"https:\/\/ajhassoc.com\/#website\"},\"datePublished\":\"2018-07-08T16:21:25+00:00\",\"author\":{\"@id\":\"https:\/\/ajhassoc.com\/#\/schema\/person\/b982087d07e1397f2767543367fb6070\"},\"breadcrumb\":{\"@id\":\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ajhassoc.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Statistical Testing is Essential to Transportation Research\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ajhassoc.com\/#website\",\"url\":\"https:\/\/ajhassoc.com\/\",\"name\":\"QRS II Page\",\"description\":\"Information about the travel forecasting package, Quick Response System II\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ajhassoc.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/ajhassoc.com\/#\/schema\/person\/b982087d07e1397f2767543367fb6070\",\"name\":\"horowitz\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/ef1125ea06cd56c8e360e1ff9c4eaf53600e7d1a0b3cc8dfa44a8bf78396e898?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ef1125ea06cd56c8e360e1ff9c4eaf53600e7d1a0b3cc8dfa44a8bf78396e898?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ef1125ea06cd56c8e360e1ff9c4eaf53600e7d1a0b3cc8dfa44a8bf78396e898?s=96&d=mm&r=g\",\"caption\":\"horowitz\"},\"url\":\"https:\/\/ajhassoc.com\/index.php\/author\/horowitz\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Statistical Testing is Essential to Transportation Research - QRS II Page","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/","og_locale":"en_US","og_type":"article","og_title":"Statistical Testing is Essential to Transportation Research - QRS II Page","og_description":"I have lately refereed journal submissions where authors have totally evaded the concept of a statistical test while trying to demonstrate the importance of their research.\u00a0 I will give somewhat altered (to protect the identities of the authors and avoid lawsuits) examples later.\u00a0 Mainly, each of these authors has developed an innovation, compared the innovation ...","og_url":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/","og_site_name":"QRS II Page","article_published_time":"2018-07-08T16:21:25+00:00","author":"horowitz","twitter_card":"summary_large_image","twitter_misc":{"Written by":"horowitz","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#article","isPartOf":{"@id":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/"},"author":{"name":"horowitz","@id":"https:\/\/ajhassoc.com\/#\/schema\/person\/b982087d07e1397f2767543367fb6070"},"headline":"Statistical Testing is Essential to Transportation Research","datePublished":"2018-07-08T16:21:25+00:00","mainEntityOfPage":{"@id":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/"},"wordCount":976,"commentCount":0,"articleSection":["Data","Research Philosophy","Travel Modeling"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#respond"]}]},{"@type":"WebPage","@id":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/","url":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/","name":"Statistical Testing is Essential to Transportation Research - QRS II Page","isPartOf":{"@id":"https:\/\/ajhassoc.com\/#website"},"datePublished":"2018-07-08T16:21:25+00:00","author":{"@id":"https:\/\/ajhassoc.com\/#\/schema\/person\/b982087d07e1397f2767543367fb6070"},"breadcrumb":{"@id":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/ajhassoc.com\/index.php\/2018\/07\/08\/statistical-testing-is-essential-to-transportation-research\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ajhassoc.com\/"},{"@type":"ListItem","position":2,"name":"Statistical Testing is Essential to Transportation Research"}]},{"@type":"WebSite","@id":"https:\/\/ajhassoc.com\/#website","url":"https:\/\/ajhassoc.com\/","name":"QRS II Page","description":"Information about the travel forecasting package, Quick Response System II","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ajhassoc.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/ajhassoc.com\/#\/schema\/person\/b982087d07e1397f2767543367fb6070","name":"horowitz","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ef1125ea06cd56c8e360e1ff9c4eaf53600e7d1a0b3cc8dfa44a8bf78396e898?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ef1125ea06cd56c8e360e1ff9c4eaf53600e7d1a0b3cc8dfa44a8bf78396e898?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ef1125ea06cd56c8e360e1ff9c4eaf53600e7d1a0b3cc8dfa44a8bf78396e898?s=96&d=mm&r=g","caption":"horowitz"},"url":"https:\/\/ajhassoc.com\/index.php\/author\/horowitz\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8zKPW-6G","_links":{"self":[{"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/posts\/414","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/comments?post=414"}],"version-history":[{"count":1,"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/posts\/414\/revisions"}],"predecessor-version":[{"id":415,"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/posts\/414\/revisions\/415"}],"wp:attachment":[{"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/media?parent=414"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/categories?post=414"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ajhassoc.com\/index.php\/wp-json\/wp\/v2\/tags?post=414"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}