<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Reproducibility |</title><link>https://AaronScherf.github.io/tags/reproducibility/</link><atom:link href="https://AaronScherf.github.io/tags/reproducibility/index.xml" rel="self" type="application/rss+xml"/><description>Reproducibility</description><generator>HugoBlox Kit (https://hugoblox.com)</generator><language>en-us</language><lastBuildDate>Fri, 06 Nov 2026 00:00:00 +0000</lastBuildDate><image><url>https://AaronScherf.github.io/media/sharing.jpg</url><title>Reproducibility</title><link>https://AaronScherf.github.io/tags/reproducibility/</link></image><item><title>Compound Shocks Panel</title><link>https://AaronScherf.github.io/projects/compound_shocks/</link><pubDate>Fri, 06 Nov 2026 00:00:00 +0000</pubDate><guid>https://AaronScherf.github.io/projects/compound_shocks/</guid><description>&lt;p&gt;In places like Somalia and Sudan, a household rarely faces one shock at a time. Fighting, a failed rainy season, and a spike in grain prices can arrive together, and displacement often follows. Most data sources describe only one of these, and each is built differently: different place names, different time steps, different units, and different rules for reuse. This project builds one monthly record per district that holds all of them, and keeps the work open to checking at every step.&lt;/p&gt;
&lt;h2 id="the-problem"&gt;The problem&lt;/h2&gt;
&lt;p&gt;Studying compound shocks needs the sources to line up. That is where quiet mistakes enter: a district spelled two ways, one conflict event counted once per actor involved, a count of people present treated as if it were a count of people moving. Humanitarian early-warning work can only be as good as the table underneath it, and a reader should be able to rebuild that table and get the same numbers.&lt;/p&gt;
&lt;p&gt;My motivation is research on how overlapping shocks relate to displacement, but the first step is infrastructure: a panel I can defend line by line before I ask any question of it.&lt;/p&gt;
&lt;h2 id="what-i-built"&gt;What I built&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;A district-month panel for two countries.&lt;/strong&gt; Somalia (74 districts) and Sudan (189 localities) from January 2021 to October 2025, 58 months each. Each row is one place in one month, with:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;conflict events and reported fatalities (ACLED);&lt;/li&gt;
&lt;li&gt;monthly rainfall from satellite estimates (CHIRPS);&lt;/li&gt;
&lt;li&gt;sorghum and wheat prices (WFP market monitoring);&lt;/li&gt;
&lt;li&gt;displacement: movements in Somalia (UNHCR/NRC PRMN, through August 2023) and numbers of displaced people present in Sudan (IOM DTM).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Decisions written down, not buried in code.&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Every source has a descriptor listing the provider, what each variable means and its unit, the place and time it covers, how access works, its reuse terms, and how far it has been verified.&lt;/li&gt;
&lt;li&gt;Counts of people &lt;em&gt;present&lt;/em&gt; (a stock) and counts of people &lt;em&gt;moving&lt;/em&gt; (a flow) stay in separate columns. They answer different questions and are not combined.&lt;/li&gt;
&lt;li&gt;Where a figure is missing, it stays missing. It is never filled with zero, because &amp;ldquo;no report&amp;rdquo; and &amp;ldquo;no events&amp;rdquo; are different facts.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Safeguards for restricted data.&lt;/strong&gt; The conflict data come from an account-based service; the project keeps raw event records out of version control and marks only district-month conflict totals for sharing. Each check on those files reports counts and checksums, never records. Reuse terms differ by source; the repository records them and does not redistribute the downloads.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;One command to fetch, one to rebuild.&lt;/strong&gt; &lt;code&gt;fetch&lt;/code&gt; downloads the sources (the conflict data through the provider&amp;rsquo;s API when an account is available) and records a checksum for each file. &lt;code&gt;reproduce&lt;/code&gt; rebuilds the panel, an audit memo with a missing-data picture, and the figures, skipping steps whose inputs have not changed. Each run writes a plain-language summary of what it made and how long it took.&lt;/p&gt;
&lt;p&gt;The project uses my
framework for the descriptors, checks, and run summaries.&lt;/p&gt;
&lt;h2 id="how-well-it-works-so-far"&gt;How well it works so far&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;It rebuilds identically.&lt;/strong&gt; On 2026-11-06 I rebuilt both panels from the raw downloads and every checksum for the panel and the publishable totals matched the earlier recorded run. Somalia takes about 4 minutes and Sudan about 14, and roughly 90% of that is reading the rainfall rasters.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The automatic download matches the manual one.&lt;/strong&gt; The API download of the conflict data contained exactly the same set of events as my earlier manual export, for both countries, and the Somalia panel built from it was byte-identical.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tested on two operating systems.&lt;/strong&gt; The 199 automated tests pass, and continuous integration runs on both Linux and Windows. Windows caught a real problem: Git was rewriting line endings in test files and silently changing their checksums. A repository setting now prevents that, and a test fails if it recurs.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Documented limits.&lt;/strong&gt; Somalia&amp;rsquo;s displacement series ends in August 2023, so later months have none. Sudan&amp;rsquo;s displacement figures are stocks, not flows, and the Sudan population file covers 188 of the 189 localities. The panel records these gaps instead of hiding them.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;What it does &lt;strong&gt;not&lt;/strong&gt; do yet: it contains no findings. I have built the data, not drawn conclusions from it.&lt;/p&gt;
&lt;h2 id="what-comes-next"&gt;What comes next&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Close the known gaps.&lt;/strong&gt; Review the missing population estimate for one Sudan locality, and move the remaining source descriptors to the newer, stricter format.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Add richer sources.&lt;/strong&gt; DTM household needs assessments for Sudan (food consumption, hunger measures, reasons for displacement); cross-border and refugee movement data; and, later, satellite-derived indicators from Google Earth Engine.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Define the questions in advance.&lt;/strong&gt; Write down the compound-shock definitions and the checks before looking at any outcome, and have the data and definitions reviewed before the first analysis.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Test the template on other places.&lt;/strong&gt; The same structure should carry to other country pairs, so the next step is trying it on a new place.&lt;/li&gt;
&lt;/ul&gt;</description></item><item><title>reprokit</title><link>https://AaronScherf.github.io/projects/reprokit/</link><pubDate>Fri, 06 Nov 2026 00:00:00 +0000</pubDate><guid>https://AaronScherf.github.io/projects/reprokit/</guid><description>&lt;p&gt;A result is only worth as much as someone else&amp;rsquo;s ability to get it again. reprokit is the scaffolding I use to make that possible: a fixed way to describe sources, their reuse terms, and how they were checked, plus a command that tells a newcomer whether their setup works.&lt;/p&gt;
&lt;h2 id="the-problem"&gt;The problem&lt;/h2&gt;
&lt;p&gt;Research projects usually fail to reproduce for ordinary reasons. The data source was never recorded, or its terms of use were not checked. A step depended on a file that existed only on one laptop. The instructions assumed the reader already knew what a terminal was. And when something does break, the message is written for the person who wrote the code.&lt;/p&gt;
&lt;p&gt;I started building it while working on my
, which combines several sources with different formats and rules. That project needed proof that its outputs rebuild exactly. I wanted that proof to be reusable, so the next project starts with it instead of adding it at the end.&lt;/p&gt;
&lt;h2 id="what-i-built"&gt;What I built&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Source descriptors.&lt;/strong&gt; A structured record for each dataset: provider, variables and units, place and time coverage, how access works, reuse terms, and how far the data has been verified. A validator checks that the record is complete and consistent with the project&amp;rsquo;s settings, and that a claim like &amp;ldquo;verified on real data&amp;rdquo; has a matching verification note.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Evidence projects.&lt;/strong&gt; A second project kind for literature and desk research keeps a source manifest and checks citations against included, verified sources. &lt;code&gt;verify-sources&lt;/code&gt; checks DOI and OpenAlex identifiers online; &lt;code&gt;migrate-review&lt;/code&gt; converts an older literature review into a draft manifest for screening.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;reprokit validate&lt;/code&gt;.&lt;/strong&gt; One command that checks all of a project&amp;rsquo;s records and prints a plain explanation for anything missing. It can also warn when a source needs an account but the README does not say how to get the data by hand.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;reprokit quick-check&lt;/code&gt;.&lt;/strong&gt; A short self-test on made-up data. It reports PASS or FAIL in ordinary language, with the next command to type, and tells the reader that it does not test the real data.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Run summaries.&lt;/strong&gt; A project can write a readable file listing the rows produced, recorded time and memory per step, and checksums for its outputs, so two people can compare their results by comparing fingerprints.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A project template.&lt;/strong&gt; A repository generated with a README for non-developers (what a terminal is, what each file is for, what to do when the check fails), a protocol document, a test setup, automated checks on Linux and Windows, and a scan for accidentally committed secrets.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Small guards learned the hard way.&lt;/strong&gt; A loader for a project&amp;rsquo;s &lt;code&gt;.env&lt;/code&gt; file for account keys (values are never printed), and a Git setting that stops Windows from rewriting line endings in data files and silently changing their checksums.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="how-well-it-works-so-far"&gt;How well it works so far&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;It is in real use.&lt;/strong&gt; The compound shocks panel is built on it. Both of that project&amp;rsquo;s full rebuilds, run on 2026-11-06, finished with every checksum matching the earlier recorded run. Two more projects are in progress: Bangladesh Insurance Replication and Evidence Map; neither has reported research results here.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automated checks.&lt;/strong&gt; All 431 automated tests pass locally, and continuous integration is configured for Linux and Windows.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Clearer first steps.&lt;/strong&gt; The generated README now explains that commands go in a terminal and names the files a newcomer will meet. Windows testing found the line-ending checksum bug above, which shipped as a fix in the next release.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Honest limits.&lt;/strong&gt; The compound shocks panel is the documented real-data use so far. The memory figure in its run summary counts only the main process, so it understates heavy steps. The checks confirm that a record is complete and consistent, not that its contents are true. Older projects need a manual update to receive new template text.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Releases so far are tagged v0.1.0 through v0.4.0 on GitHub.&lt;/p&gt;
&lt;h2 id="what-comes-next"&gt;What comes next&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Try it in the two in-progress projects&lt;/strong&gt; and see what breaks beyond the compound shocks panel.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Measure memory properly&lt;/strong&gt; and say plainly what the number covers.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Make adoption easier&lt;/strong&gt; for an existing repository, including a clear note on how to generate a project from a local copy and how to repair line endings in older checkouts.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Migrate remaining records&lt;/strong&gt; to the current descriptor format so the stricter checks run everywhere.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Keep reviewing it with fresh eyes&lt;/strong&gt;, especially the first ten minutes for a reader who has never opened a terminal.&lt;/li&gt;
&lt;/ul&gt;</description></item></channel></rss>