Limitations and responsible usage
gst1.org is a tool for exploring the 200,000 pages of submissions to the Global Stocktake. We have created it to facilitate search and analysis of the submissions and to gather data on the value to our users of our policy concept identification processes and the way in which we make use of them.
It is an R&D preview. This means that it has not gone through the same quality assurance and improvement processes that our products usually do. We do this so that we can get tools out to the community quickly, but it means that there are limitations that you should be aware of.
Responsible usage
It is OK to use gst1.org for tasks like:
- Finding examples of language relating to a policy concept
- Finding where submissions mention a specific term, without relying on it as an authoritative final answer
- Gathering evidence to support or invalidate hypotheses in an analysis, as long as it is OK if not all mentions are found
- As a starting point to help identify documents and places within documents to continue your research
You must take care when using gst1.org for tasks like:
- Determining exhaustively if a concept is or is not mentioned, and any further analysis of the distribution of those mentions
- Landscape analysis, or analysis that relies on an exhaustive view of all mentions of a document
- Any analysis that takes absence of a mention as evidence
Document library
We have automatically scraped all the documents from several UNFCCC portals to populate the document library. This is done on a schedule and is a fragile process (for example, small changes made by the UNFCCC can cause errors in our scripts), meaning that not all documents may be present. We then extract the full text from these documents. This also can create errors, where the document has been created in a particularly inaccessible form to our code. At present, ~1% of documents may be missing or with missing/corrupted text (full known list here).
Policy concepts
We have created a taxonomy of policy-relevant concepts - the thematic filters on the left hand side - that help users find passages that mention a specific concept. For example, selecting the *Fossil Fuels* thematic filter will return passages that our automations have identified as relating to fossil fuels. These automations are based on a variety of language processing techniques, but represent a first iteration of their application. As such, the performance varies. Our roadmap activities will improve the performance over time.
Performance with these sorts of techniques can be measured by the number of:
- ‘false positives’ — how many things you will see labelled as “X” that aren’t X
- ‘false negatives’ — how many things *should* have been labelled as “X” but weren’t, and therefore will not be in your filtered search results.
The rates of false positives and false negatives we’ve measured per concept varies widely, and we are now working on improving these, but on average we are identifying ~60% of mentions of concepts and ~2/3 of the mentions that we identify are correct. This means that search results you see may not match the filter you have selected all the time, and you should not assume you are seeing all mentions of the policy concept that exist in the document library.