Jekyll2022-03-01T15:22:13+00:00https://europepmc.github.io/techblog/feed.xmlEurope PMC Tech BlogTechy stuff from Europe PMCEurope PMC Text-annotator is now open source2020-11-02T00:00:00+00:002020-11-02T00:00:00+00:00https://europepmc.github.io/techblog/literature_data_integration/2020/11/02/europe-pmc-text-annotator-is-now-open-source<style type="text/css" scoped="">
a img {
border: 2px solid #ccc;
}
</style>
<h2 id="introducing-text-annotator">Introducing <em>Text-annotator</em></h2>
<p>Europe PMC has open-sourced <em>Text-annotator</em>, a JavaScript library to locate and annotate plain text in HTML.
The annotation process includes:</p>
<ol>
<li><strong>Search</strong>: Search for a piece of plain text in the HTML; on finding it, store its location identified by an index and then return the index for later annotation.<!--more--></li>
<li><strong>Annotate</strong>: Annotate the found text given its index. <br /><br />
In order to annotate a piece of text, two steps, search and annotate, are taken. The idea of splitting the annotation process into the two steps is to allow more flexibility, e.g., the user can search for all pieces of text first and then annotate them later as required. <em>Text-annotator</em> can be used in the browser or the Node.js server.</li>
</ol>
<h2 id="how-is-text-annotator-used-in-europe-pmc">How is <em>Text-annotator</em> used in Europe PMC?</h2>
<p>The <em>Text-annotator</em> has been used in <a href="https://europepmc.org">Europe PMC</a> in the following features:</p>
<h3 id="article-title-highlighting">Article title highlighting</h3>
<p>When searching in Europe PMC, the search term will be highlighted in the title of articles displayed in the search result page.</p>
<p><a href="/techblog/images/posts/text-annotator-open-source/image4.png"><img src="/techblog/images/posts/text-annotator-open-source/image4.png" alt="Search example" /></a>
<small style="display:block;text-align:center;"><a href="https://europepmc.org/search?query=p53">https://europepmc.org/search?query=p53</a></small></p>
<h3 id="snippets">Snippets</h3>
<p><em>Text-annotator</em> also supports the snippets, which are highlights from the article matching the searched query.</p>
<p><a href="/techblog/images/posts/text-annotator-open-source/image5.png"><img src="/techblog/images/posts/text-annotator-open-source/image5.png" alt="Snippets example" /></a>
<small style="display:block;text-align:center;"><a href="https://europepmc.org/article/MED/33121131">https://europepmc.org/article/MED/33121131</a></small></p>
<p>Every search result displays two snippets, separated by an ellipsis. The following shows the snippets in an article returned for a search term ‘cancer’</p>
<p><a href="/techblog/images/posts/text-annotator-open-source/image1.png"><img src="/techblog/images/posts/text-annotator-open-source/image1.png" alt="Search result snippets example" /></a>
<small style="display:block;text-align:center;"><a href="https://europepmc.org/search?query=cancer">https://europepmc.org/search?query=cancer</a></small></p>
<h3 id="scilite">SciLite</h3>
<p>The <em>Text-annotator</em> was also used for the development of <a href="http://blog.europepmc.org/2020/09/announcing-new-version-of-scilite.html">SciLite</a>, a tool for highlighting biological entities, such as genes/proteins, accession numbers, protein interactions, diseases, gene-disease relationship, in the full text of life sciences articles in Europe PMC. This function combines the two steps of <em>Text-annotator</em>, search and annotation. SciLite currently provides access to over 1.3 billion annotations in articles.</p>
<p><a href="/techblog/images/posts/text-annotator-open-source/image2.png"><img src="/techblog/images/posts/text-annotator-open-source/image2.png" alt="Scilite example" /></a>
<small style="display:block;text-align:center;"><a href="https://europepmc.org/article/PMC/PMC6423025">https://europepmc.org/article/PMC/PMC6423025</a></small></p>
<h3 id="linkback">Linkback</h3>
<p>When using SciLite, the highlighted term will provide a popup window with the name of the annotation provider and a link to the data resource. The feature that allows users to get the linkback to this annotation via the ‘Share’ option also is supported by <em>Text-annotator</em>.</p>
<p><a href="/techblog/images/posts/text-annotator-open-source/image3.png"><img src="/techblog/images/posts/text-annotator-open-source/image3.png" alt="Linkback example" /></a>
<small style="display:block;text-align:center;"><a href="http://europepmc.org/article/PMC/PMC6423025#europepmc-b1865e80dfcfdd5d919150aaa56e7b0b">http://europepmc.org/article/PMC/PMC6423025#europepmc-b1865e80dfcfdd5d919150aaa56e7b0b</a></small></p>
<p>The source code for <em>Text-annotator</em> is available on <a href="https://gitlab.ebi.ac.uk/literature-services/public-projects/text-annotator">Gitlab</a>. We welcome feedback and hope that you find it useful! Check out some of the other projects we’ve open-sourced at <a href="https://gitlab.ebi.ac.uk/literature-services/public-projects">GitLab</a> and <a href="https://github.com/EuropePMC">Github</a>.</p>
<p>For more information, contact <a href="mailto:helpdesk@europepmc.org">helpdesk@europepmc.org</a>.</p>["dayane", "zhan"]Introducing Text-annotator Europe PMC has open-sourced Text-annotator, a JavaScript library to locate and annotate plain text in HTML. The annotation process includes: Search: Search for a piece of plain text in the HTML; on finding it, store its location identified by an index and then return the index for later annotation.Europe PMC API use cases page2020-01-28T00:00:00+00:002020-01-28T00:00:00+00:00https://europepmc.github.io/techblog/literature_data_integration/2020/01/28/europe-pmc-api-use-cases-page<h2 id="connecting-the-programmatic-user-community">Connecting the programmatic user community</h2>
<p>Europe PMC makes its open access content and metadata available for building new applications. With a <a href="http://europepmc.org/developers">developer tab</a> specially designed for programmatic users and with detailed documentation, the aim is to make it easy to integrate Europe PMC APIs into new services and thereby improve life science research.
<!--more--></p>
<p>Programmatic users can now flag the applications they have created using Europe PMC APIs in the newly released <a href="http://europepmc.org/use-cases">API use cases page</a>. This new page aims to foster interaction within the Europe PMC API user community, connecting programmatic users and allowing them to share their work and improve the usability of their open source code.</p>
<p>Do read these testimonials and see what nuggets you can glean from other users. If you have pearls of wisdom to pass on or would simply like to flag your integration and membership of this user community, please contact us on <a href="mailto:helpdesk@europepmc.org">helpdesk@europepmc.org</a>.</p>
<p><img src="/techblog/images/posts/use_case_page/Use_case_image.png" style="border: 1px solid #ccc" alt="API use cases page" /></p>["dayane"]Connecting the programmatic user community Europe PMC makes its open access content and metadata available for building new applications. With a developer tab specially designed for programmatic users and with detailed documentation, the aim is to make it easy to integrate Europe PMC APIs into new services and thereby improve life science research.New release: Europe PMC Web Services 6.32020-01-28T00:00:00+00:002020-01-28T00:00:00+00:00https://europepmc.github.io/techblog/literature_data_integration/2020/01/28/new-release-europe-pmc-web-services<p>Today Europe PMC released Web Services version 6.3, updating its SOAP and RESTful APIs. Programmatic users can now find more fields available in the core response of these services</p>
<h2 id="multiple-author-affiliations">Multiple author affiliations</h2>
<p>In the previous version, author affiliations were a single value. The new version 6.3 includes a list of multiple affiliations.</p>
<!--more-->
<p>Example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><authorList>
<author>
<fullName>Wei W</fullName>
<firstName>Wei</firstName>
<lastName>Wei</lastName>
<initials>W</initials>
<authorAffiliationsList>
<authorAffiliation>
Department of Orthopaedics, The 1st Affiliated Hospital of Harbin Medical University, Harbin, China.
</authorAffiliation>
<authorAffiliation>
Department of Orthopaedics, Harbin 242 Hospital, Harbin, China.
</authorAffiliation>
</authorAffiliationsList>
</author>
</authorList>
</code></pre></div></div>
<h2 id="datalinks-tags">Datalinks tags</h2>
<p>Articles now also contain data links tags, which can be used to retrieve data links using a tag parameter.</p>
<p>Example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dataLinksTagsList>
<dataLinkstag>altmetrics</dataLinkstag>
<dataLinkstag>supporting_data</dataLinkstag>
</dataLinksTagsList>
</code></pre></div></div>
<p>Check the <a href="https://europepmc.org/RestfulWebService">Articles RESTful API</a> page to find more information on Europe PMC data links.</p>
<h2 id="publication-status">Publication status</h2>
<p>Retrieving the status of the article is now also possible. The new field ‘Publication status’ indicates, for example, if a given article is published or ahead of print.</p>
<p>The possible values for publication status are:</p>
<dl>
<dt>received</dt> <dd>date manuscript received for review</dd>
<dt>accepted</dt> <dd>accepted for publication</dd>
<dt>epublish</dt> <dd>published electronically by publisher</dd>
<dt>ppublish</dt> <dd>published in print by publisher</dd>
<dt>revised</dt> <dd>article revised by publisher/author</dd>
<dt>pmc</dt> <dd>article first appeared in PubMed Central</dd>
<dt>pmcr</dt> <dd>article revision in PubMed Central</dd>
<dt>pubmed</dt> <dd>article citation first appeared in PubMed</dd>
<dt>pubmedr</dt> <dd>article citation revision in PubMed</dd>
<dt>aheadofprint</dt> <dd>epublish, but will be followed by print</dd>
<dt>premedline</dt> <dd>date into PreMedline status</dd>
<dt>medline</dt> <dd>date made a MEDLINE record</dd>
<dt>other</dt><dd></dd>
</dl>
<p>Example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><publicationStatus>ppublish</publicationStatus>
</code></pre></div></div>
<p>Let us know your thoughts and questions regarding this new release. Contact <a href="mailto:helpdesk@europepmc.org">helpdesk@europepmc.org</a> or post on the <a href="https://groups.google.com/a/ebi.ac.uk/forum/#!forum/epmc-webservices">Europe PMC developer forum</a>.</p>["dayane", "mohamed"]Today Europe PMC released Web Services version 6.3, updating its SOAP and RESTful APIs. Programmatic users can now find more fields available in the core response of these services Multiple author affiliations In the previous version, author affiliations were a single value. The new version 6.3 includes a list of multiple affiliations.PDBe integrates Europe PMC APIs2019-10-16T00:00:00+00:002019-10-16T00:00:00+00:00https://europepmc.github.io/techblog/literature_data_integration/2019/10/16/using-europe-pmc-restful-apis<h2 id="pdbe-exposes-literature-metadata-by-integrating-europe-pmc-rest-apis">PDBe exposes literature metadata by integrating Europe PMC REST APIs</h2>
<p><a href="https://www.ebi.ac.uk/pdbe/">PDBe</a>, a member of the Worldwide Protein Bank, is a European resource that maintains a free and publicly available archive of macromolecular structures. The public can easily find information on protein structure and metadata associated with protein structure. PDBe also exposes enriched metadata from other sources such as publications and citations, which are retrieved through Europe PMC APIs.
<!--more--></p>
<p>By integrating knowledge from Europe PMC open-access publications, PDBe provides entries with 3D macromolecular structures in context with literature highlighting the impact of a given structure on scientific research. The first step in the integration process is to map the PDB IDs to PubMed IDs (e.g PDB “1cbs” has “PMID7704533” as its primary publication). The second step is to use the Europe PMC REST API <a href="https://europepmc.org/RestfulWebService#!/Europe32PMC32Articles32RESTful32API/search">GET /search</a> request to retrieve the publications by PMID, which will populate a PDBe page with fields such as publication DOI, title, authorList, journal and abstract.</p>
<p><a href="/techblog/images/posts/using-europe-pmc-restful-apis/Blog_image01.png"><img src="/techblog/images/posts/using-europe-pmc-restful-apis/Blog_image01.png" alt="Illustration showing how the webservices response becomes text on a webpage in PDBe" /></a>
<small style="display:block; text-align: right;"><em><strong>Figure 1</strong>: <a href="https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=7704533&resultType=core&cursorMark=*&pageSize=25&format=json">Europe PMC’s search API response</a> becomes <a href="https://www.ebi.ac.uk/pdbe/entry/pdb/1cbs/">a page on PDBe</a></em></small></p>
<p>PDBe also retrieves citation counts through the Europe PMC REST API <a href="https://europepmc.org/RestfulWebService#!/Europe32PMC32Articles32RESTful32API/citations">GET /citations</a> request for each PMID. The response is manipulated in order to display the number and links to reviews and experimental articles citing a given publication.</p>
<p><a href="/techblog/images/posts/using-europe-pmc-restful-apis/Blog_image02.png"><img src="/techblog/images/posts/using-europe-pmc-restful-apis/Blog_image02.png" alt="Illustration showing how the webservices response becomes text on a webpage in PDBe" /></a>
<small style="display:block; text-align: right;"><em><strong>Figure 2</strong>: <a href="https://www.ebi.ac.uk/europepmc/webservices/rest/MED/7704533/citations?page=1&pageSize=25&format=json">Europe PMC’s citations API response</a> becomes <a href="https://www.ebi.ac.uk/pdbe/entry/pdb/1cbs/citations">a page on PDBe</a></em></small></p>
<p>The integration of Europe PMC APIs in PDBe web services was very simple. Once we knew what data we were interested in, we just needed to use the correct API endpoint with the corresponding parameters to populate PDBe database with data gleaned from the response. This data is then used for PDBe pages and PDBe REST API providing richer context to protein structure records.</p>["dayane", "nurul"]PDBe exposes literature metadata by integrating Europe PMC REST APIs PDBe, a member of the Worldwide Protein Bank, is a European resource that maintains a free and publicly available archive of macromolecular structures. The public can easily find information on protein structure and metadata associated with protein structure. PDBe also exposes enriched metadata from other sources such as publications and citations, which are retrieved through Europe PMC APIs.Europe PMC project for eLife Innovation Sprint2019-08-20T00:00:00+00:002019-08-20T00:00:00+00:00https://europepmc.github.io/techblog/literature_data_integration/2019/08/20/elife-innovation-sprint<h2 id="how-to-find-the-perfect-preprint">How to find the perfect preprint</h2>
<p>The <a href="https://sprint.elifesciences.org/">eLife Innovation Sprint</a> is a yearly collaborative hackathon for developers, designers, researchers, technologists, science communicators, and everyone enthusiastic about open science. The premise of the Sprint is simple – the current science publishing system is slow, inefficient and insanely expensive. What we need are open science ideas that could be turned into prototypes to address the challenges we face in science publishing. All Sprint outputs have to be openly available, use open-source licenses for code and software, and permissive licenses (such as CC-BY) for other content.</p>
<p>This year the Europe PMC team will participate in the Sprint with a proposal to improve the discoverability of relevant scientific preprints. <a href="https://www.ebi.ac.uk/about/people/dayane-rodrigues-araujo">Dayane Araujo</a> (Technical Outreach Officer) and <a href="https://www.ebi.ac.uk/about/people/michael-parkin">Michael Parkin</a> (Data Scientist) will be working on a tool to sort through ~80,000 life science preprints, and they need your help.</p>
<!--more-->
<h2 id="project-idea">Project idea</h2>
<p>Considering how the acceptance of preprints in the biomedical field has increased, how do researchers come across relevant preprints when looking for what is new in their scientific field? Preprints rarely have associated keywords to make filtering easier. The majority of preprints lack full-text available for deep indexing. Since preprints appear hot-off-the-press, they often don’t accumulate many citations, and are not associated with journal impact factors, commonly used as a proxy for scientific quality. This makes it harder to decide what preprint to read next.</p>
<p>What if there was a tool to find and sort preprints by topic? The tool could visualise which preprints have been commented on, most read, and/or most downloaded, to help with the selection process. Such a tool would help improve preprint discoverability.</p>
<p><a href="https://europepmc.org/">Europe PMC</a>, a database of research literature, indexes preprints from multiple sources, in addition to traditional journal publications. We will use the <a href="https://europepmc.org/AnnotationsApi">Europe PMC Annotations API</a> to retrieve text-mined biological concepts from preprint abstracts, sorting preprints into categories. We will track preprint reviews linked to preprints in Europe PMC. Additionally, we will use Crossref and other sources to retrieve comments, number of downloads, social media mentions, and other associated metrics.
Our resulting open source package will help select preprints by topic, and display comments, recommendations, and associated preprint metrics. This package could be re-used by preprint servers to provide topic streams, by preprint journal clubs and recommendation platforms to select content for review, and by researchers to simplify preprint discovery process.</p>
<h2 id="how-can-you-help">How can you help</h2>
<p>This project will benefit from team members with experience using REST APIs, particularly Crossref event data service. It will be essential to get user insights from those reading, posting and commenting on preprints, as well as those selecting preprints for journal clubs.</p>
<p>If you would like to help us take preprints to the next level, <a href="mailto:dayane@ebi.ac.uk">please get in touch with Dayane</a>. And for all participants at the eLife Innovation Sprint 2019 interested in preprinting – we look forward to working together!</p>Maria LevchenkoHow to find the perfect preprint The eLife Innovation Sprint is a yearly collaborative hackathon for developers, designers, researchers, technologists, science communicators, and everyone enthusiastic about open science. The premise of the Sprint is simple – the current science publishing system is slow, inefficient and insanely expensive. What we need are open science ideas that could be turned into prototypes to address the challenges we face in science publishing. All Sprint outputs have to be openly available, use open-source licenses for code and software, and permissive licenses (such as CC-BY) for other content. This year the Europe PMC team will participate in the Sprint with a proposal to improve the discoverability of relevant scientific preprints. Dayane Araujo (Technical Outreach Officer) and Michael Parkin (Data Scientist) will be working on a tool to sort through ~80,000 life science preprints, and they need your help.A perfect match: locating plain text in HTML pages2018-07-04T00:00:00+00:002018-07-04T00:00:00+00:00https://europepmc.github.io/techblog/algorithm/2018/07/04/locating-text-html-pages<p><a href="https://europepmc.org/Annotations">SciLite</a> is a Europe PMC tool that allows biological terms or relations, such as diseases, chemicals or protein interactions, to be highlighted for readers on abstracts and full text articles. These terms are identified as annotations by text mining algorithms, developed by a variety of text mining groups.</p>
<p>The main challenge for the SciLite tool is locating plain text annotations in HTML pages. The challenges derive from the nature of HTML pages. Below is a list of the major challenges we faced and the solutions adopted to mitigate them.</p>
<ol>
<li>
<p><strong>The pages contain HTML tags, obviously.</strong> For example, <a href="https://europepmc.org/articles/PMC1215513">visit this article</a>, and click on the “Gene Function” checkbox, on the right-hand side of the page, to see the sentence highlighted. <br /><br />
<a href="/techblog/images/posts/locating-text-html-pages/fuzzy_match_PMC1215513.png"><img src="/techblog/images/posts/locating-text-html-pages/fuzzy_match_PMC1215513.png" alt="Annotation containing HTML tags" /></a>
<em><strong>Figure 1</strong>: Annotation containing HTML tags</em><br /><br /><!--more-->
The problem is caused by the <code class="language-plaintext highlighter-rouge"><sub></code> tag that surrounds the character “v” inside the word “Nav1.7”. Thus if you search for an exact match of the plain sentence in the HTML page, it will not be found. Our solution was to search for a regular expression built to include an optional HTML tag between any two characters of the annotation text. The disadvantage of this approach is that this type of search is much more computationally demanding than an exact match search. Therefore, we decided to adopt this regular expression search only for sentence-based annotations, where the chance of having HTML tags is much higher than for named-entity annotations (usually composed of only one or two words).</p>
</li>
<li>
<p><strong>HTML encodes special characters.</strong> An example is the character “>”. It is encoded as <code class="language-plaintext highlighter-rouge">&gt;</code> inside an HTML page. For example, <a href="http://europepmc.org//abstract/MED/28385055">visit this article</a>, and click on the “Gene Disease Open Targets” checkbox.<br /><br />
<a href="/techblog/images/posts/locating-text-html-pages/fuzzy_match_MED28385055.png"><img src="/techblog/images/posts/locating-text-html-pages/fuzzy_match_MED28385055.png" alt="Annotation containing HTML encoded characters" /></a>
<em><strong>Figure 2</strong>: Annotation containing HTML-encoded characters</em><br /> <br />
The text of the annotation contains the character “>”. Our solution was to encode the annotation text as it would appear in an HTML page (replacing > with <code class="language-plaintext highlighter-rouge">&gt;</code>), and then perform an exact match search.</p>
</li>
<li>
<p><strong>Lack of correspondence between the text of the annotation and text inside the HTML.</strong> For example, <a href="http://europepmc.org/articles/PMC3558359">visit this article</a>, and click on the “Gene Function” checkbox.<br /><br />
<a href="/techblog/images/posts/locating-text-html-pages/fuzzy_match_PMC3558359.png"><img src="/techblog/images/posts/locating-text-html-pages/fuzzy_match_PMC3558359.png" alt="Annotation containing Greek characters" /></a>
<em><strong>Figure 3</strong>: Annotation containing Greek characters</em><br /><br />
The original annotation text is “Our results revealed a direct interaction between PRL-3 and integrin beta1 and characterized Y783 of integrin beta1 as a bona fide substrate of PRL-3, which is negatively regulated by integrin alpha1.” The problem is that the Greek letters alpha and beta are represented in two different ways in the page and in the annotation text. A solution to this problem is applying a fuzzy match approach that is discussed later.</p>
</li>
<li>
<p><strong>Special characters are not properly encoded inside the annotation text.</strong> For example, <a href="http://europepmc.org/abstract/AGR/IND605699789">visit this article</a>, click on the “Organism” checkbox, and focus on the annotation “white campion”.<br /><br />
<a href="/techblog/images/posts/locating-text-html-pages/fuzzy_match_AGRIND605699789.png"><img src="/techblog/images/posts/locating-text-html-pages/fuzzy_match_AGRIND605699789.png" alt="Annotation containing not properly encoded characters" /></a>
<em><strong>Figure 4</strong>: Annotation containing not properly encoded characters</em><br /><br />
Every annotation comes with prefix and suffix text that help to locate it in the article page. The suffix of this annotation is “is subject to preâdispersal” with the character <code class="language-plaintext highlighter-rouge">&acirc;</code> not properly encoded. In this case as well, our solution was to apply our fuzzy match approach.</p>
</li>
</ol>
<h2 id="fuzzy-match-strategy">Fuzzy Match Strategy</h2>
<p>The fuzzy match approach we used to solve some of the problems described above is based on the open source Javascript library <a href="http://fusejs.io/">Fuse.js</a>. Internally it uses the <a href="https://en.wikipedia.org/wiki/Levenshtein_distance">Levenshtein distance</a> to compute the similarity score between two strings. This score is computed as the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.</p>
<p>The approach consists of the following steps:</p>
<ol>
<li>The article text is split into sentences after stripping all the HTML tags from it (using an open access <a href="https://github.com/Tessmore/sbd">javascript sentencizer</a>).</li>
<li>The fuzzy match algorithm compares the annotation sentences with the list of sentences inside the article. A similarity threshold is defined in order to retrieve only the article sentences that are similar enough to the annotation text. The choice of this threshold is crucial. If it is too restrictive, there is a chance that some true positive matches will be ignored, while if it is too lax, there is a chance that some false positive matches will be found.</li>
<li>If there is at least one sentence considered similar enough to the annotation, the sentence with the highest similarity score is taken as a valid match.</li>
</ol>
<p>As has been discussed previously, there are two type of annotations inside the SciLite platform:</p>
<ul>
<li>Sentence-based annotations.</li>
<li>Named-entity annotations (usually consisting of one/two words) with a prefix and a suffix to locate them inside the article.</li>
</ul>
<p>Because of the nature of the fuzzy match algorithm, it can be directly applied only to sentence-based annotations. Since the fuzzy match search is more computationally demanding than an exact search, we decided to adopt it only if the exact search for the sentence fails.</p>
<p>The fuzzy match has also proven useful for matching the prefix and the suffix of the named entity annotations, as described in problem number 4. Even in this case, we adopt a fuzzy match prefix/suffix search only if the exact search fails, to avoid computational overhead.</p>
<p>We have run some tests to compare the numbers of annotations matched with and without this fuzzy match approach. The sample was made of 8433 full text articles plus 8368 abstracts. The results are as follows:</p>
<table>
<thead>
<tr>
<th></th>
<th>Fuzzy Match</th>
<th>Exact Match</th>
</tr>
</thead>
<tbody>
<tr>
<td>Sentence based</td>
<td>91.62 %</td>
<td>79.1 %</td>
</tr>
<tr>
<td>Named Entity</td>
<td>92.11 %</td>
<td>86.93 %</td>
</tr>
</tbody>
</table>
<p><em><strong>Table 1</strong>: Fuzzy match approach results</em><br /></p>
<p>As expected, you can see that the benefits of the fuzzy match approach are more significant for sentence-based annotations.</p>
<h2 id="conclusions">Conclusions</h2>
<p>Searching for plain text in HTML pages presents many challenges due to the nature of HTML rendering (tags, characters encoding, mismatched characters). One approach to solve these issues is to introduce techniques that apply some sort of fuzzy matching. However, those techniques can be demanding from a performance point of view, especially if the HTML pages are long and the number of annotations to locate is large. Therefore, it is necessary to carefully balance accuracy of results and performance, deciding when it is appropriate to apply those strategies.</p>Francesco Talo'SciLite is a Europe PMC tool that allows biological terms or relations, such as diseases, chemicals or protein interactions, to be highlighted for readers on abstracts and full text articles. These terms are identified as annotations by text mining algorithms, developed by a variety of text mining groups. The main challenge for the SciLite tool is locating plain text annotations in HTML pages. The challenges derive from the nature of HTML pages. Below is a list of the major challenges we faced and the solutions adopted to mitigate them. The pages contain HTML tags, obviously. For example, visit this article, and click on the “Gene Function” checkbox, on the right-hand side of the page, to see the sentence highlighted. Figure 1: Annotation containing HTML tagsIntegrating Literature and Data2018-06-04T00:00:00+00:002018-06-04T00:00:00+00:00https://europepmc.github.io/techblog/literature_data_integration/2018/06/04/integrating-literature-and-data<p>Data is at the heart of research. Scientific papers describe how data has been obtained, analysed, and what conclusions have been drawn. But it is the data that comprises the essential evidence, which confirms or disproves the original hypothesis. In the life sciences it is essential to look at scientific literature in the context of other publications, the data it builds on and other data linked to the publication. At Europe PMC we have developed a number of features to support data discovery and reuse.</p>
<p>As one of the <a href="https://www.google.com/url?q=https://europepmc.org/articles/PMC5070591&sa=D&ust=1527611404257000&usg=AFQjCNGDEarF0yK0Ktref2C7QmlslyDnAw">ELIXIR Core Resources</a>, Europe PMC benefits from excellent links to essential research data hubs located at EMBL-EBI. This helps us interweave publications and data, enriching the graph of research objects, and help researchers discover linked and related data.</p>
<p>The literature-data links come in different forms and shapes. An article might be citing a DOI for a dataset in a repository, or describe a protein structure cited as an accession number for PDBe database. An publication itself might be cited by a database, such as Flybase or even a Wikipedia article. Europe PMC obtains such literature-data links in three ways: <!--more-->they might be provided directly by databases at the EBI; submitted by providers participating in Europe PMC <a href="http://europepmc.org/LabsLink">external links program</a>; or picked up directly by our text-mining pipeline that extracts data mentions, such as accession numbers, from research publications. The three link types largely share the same characteristics, but used to be scattered through the Europe PMC website. They would show up in different locations and were obtained through different web service methods in different formats, even though they all link external content, mostly data, to the publication.</p>
<p>Based on their commonalities and use, it made sense for us to start consolidating our datalinks in the Europe PMC API, as well as in their presentation through the Europe PMC website. To adhere to community standards and allow exchange of data with other providers, we have turned to the Scholix format for scholarly link exchange, which we have helped to shape and have subsequently used to represent datalinks in Europe PMC web services.</p>
<h2 id="collaborating-with-scholix">Collaborating with Scholix</h2>
<p><a href="http://scholix.org/">Scholix</a>, or <strong>Scho</strong>larly <strong>li</strong>nk e<strong>x</strong>change, is an initiative is to establish a multi-hub infrastructure to harmonize and enable the exchange of data-literature links between several natural hubs, such as DataCite, CrossRef, or OpenAIRE, in scholarly communities.
The centerpiece of the Scholix landscape is the <a href="http://www.scholix.org/schema">format</a> that is used to facilitate link exchange between the hubs and other interested parties. Data links in Scholix format are presented as an “information package”. The package contains information about the two linked objects (e.g. a publication and a dataset), as well as link metadata: date, provider, copyrights, etc.</p>
<p><img src="/techblog/images/posts/integrating-literature-and-data/SCHOLIX.png" alt="Scholix Hub Architecture" />
<em><strong>Figure 1</strong>: Scholix hub architecture</em></p>
<p>Europe PMC is a part of the Research Data Alliance World Data System (RDA/WDS) <a href="https://www.rd-alliance.org/groups/rdawds-scholarly-link-exchange-scholix-wg">Working Group on Scholarly Link Exchange</a> that has pioneered the Scholix format and the multi hub approach. To consolidate internal data link formats and simplify exchange with external partners we have built the Europe PMC API method for data links around the Scholix format.</p>
<h2 id="implementing-scholix-for-europe-pmc">Implementing Scholix for Europe PMC</h2>
<p>When implementing Scholix format for literature-data links in Europe PMC we had to take into account specific requirements imposed by Europe PMC front end. The Scholix information package provides a core set of metadata, which we have supplemented with custom elements. As mentioned before, data links in Europe PMC come in through three main routes and have some unique specifications. For example, the information package for text-mined accessions in addition to the standard Scholix metadata will include the number of occurrences in text, while a link submitted via External links program might contain an image - e.g. metrics information provided by Altmetrics. In order to preserve this additional information we have embedded Scholix link information package into a hierarchy of categories. One category corresponds to a single resource which hosts the data linked to a publication. For each category (e.g. Uniprot, Wikipedia, Altmetrics) there might be multiple links within the article. Those links are further categorised as sections, depending on the method, which we used to obtain it. Consequently, there are three available sections:
Text-mined links (Biological Entities and Accessions of such, discovered through our text-mining pipeline)
Database cross-references (Links to EBI resources submitted directly to Europe PMC)
External links (Links submitted through the Europe PMC <a href="http://europepmc.org/LabsLink">external links program</a>. This is a mixed set, which includes database records, as well as links to lay summaries, press releases and open peer reviews)</p>
<p><img src="/techblog/images/posts/integrating-literature-and-data/Swagger.png" alt="Interactive API Documentation using Swagger" />
<em><strong>Figure 2</strong>: Interactive swagger documentation of the datalinks web service method</em></p>
<p>The categories and sections are reflected in Europe PMC Swagger-powered RESTful <a href="http://europepmc.org/RestfulWebService#meths">API documentation</a>. If you want to try it out, query <a href="https://www.ebi.ac.uk/europepmc/webservices/rest/MED/28818901/datalinks?format=json">https://www.ebi.ac.uk/europepmc/webservices/rest/MED/28818901/datalinks?format=json</a> to retrieve all links associated with the publication PMID:28818901 in JSON format.</p>
<p>For Europe PMC API users, or those who are planning to give the Europe PMC API a go, we recommend signing up for Europe PMC <a href="https://groups.google.com/a/ebi.ac.uk/forum/#!forum/epmc-webservices">developer forum</a>. We are keeping subscribers up to speed with changes and help out if there are any questions regarding Europe PMC APIs. You may also find like-minded peers using the datalinks API method already.</p>
<p>For the front end users this means having all data links conveniently combined in a single place on the article page - the Data tab. It contains links to supplemental files hosted by the BioStudies database, as well as related data, or data cited in the article.</p>
<p><img src="/techblog/images/posts/integrating-literature-and-data/data-tab-scr.png" alt="Europe PMC Data Tab Example" />
<em><strong>Figure 3</strong>: The new Europe PMC data tab</em></p>
<h2 id="to-wrap-up">To wrap up</h2>
<p>Overall the new data links module not only powered a new user-facing feature (the Data tab), but it also consolidated three existing API methods related to data-literature links into one, simplifying and enabling major front end developments in the future.
Furthermore, using the Scholix format helps make literature-data links available and accessible to data link hubs like Datacite, CrossRef, and OpenAIRE, as well as to individual data scientists using Europe PMC.</p>Florian GräfData is at the heart of research. Scientific papers describe how data has been obtained, analysed, and what conclusions have been drawn. But it is the data that comprises the essential evidence, which confirms or disproves the original hypothesis. In the life sciences it is essential to look at scientific literature in the context of other publications, the data it builds on and other data linked to the publication. At Europe PMC we have developed a number of features to support data discovery and reuse. As one of the ELIXIR Core Resources, Europe PMC benefits from excellent links to essential research data hubs located at EMBL-EBI. This helps us interweave publications and data, enriching the graph of research objects, and help researchers discover linked and related data. The literature-data links come in different forms and shapes. An article might be citing a DOI for a dataset in a repository, or describe a protein structure cited as an accession number for PDBe database. An publication itself might be cited by a database, such as Flybase or even a Wikipedia article. Europe PMC obtains such literature-data links in three ways:The Importance of Software Testing in DevOps2018-03-08T00:00:00+00:002018-03-08T00:00:00+00:00https://europepmc.github.io/techblog/testing/2018/03/08/importance-of-software-testing-in-ci-cd<p>Software testing is the process of identifying the correctness and quality of a software program. In other words, testing is executing a system or application in order to find software bugs, defects, errors or unexpected behavior.</p>
<p>Software testing is necessary because we all make mistakes. Some of those mistakes are minor, but others can be expensive or dangerous. Especially while practicing <a href="https://www.atlassian.com/continuous-delivery/ci-vs-ci-vs-cd">continuous integration, continuous delivery, or continuous deployment</a>, we need to test anything and everything we produce, because things can always go wrong.</p>
<p>Testing is mainly classified into two types, <strong>Functional Testing</strong> and <strong>Non-functional Testing</strong>.</p>
<p><a href="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST1.JPG"><img src="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST1.JPG" alt="Types of Functional and Non-functional testing" /></a></p>
<!--more-->
<p>Functional testing processes test against the specifications, or functional requirements for a system and its components (what the system should <em>do</em>), while non-functional testing processes test the operation of that system (how the system should <em>be</em>).</p>
<h2 id="common-types-of-functional-testing">Common types of Functional Testing</h2>
<h3 id="unit-testing">Unit Testing</h3>
<p>In Unit Testing, the scope of testing is limited to a particular unit or component—for example, a specific <a href="https://en.wikipedia.org/wiki/Application_programming_interface">API</a>, <a href="https://en.wikipedia.org/wiki/Database">database</a>, or <a href="https://en.wikipedia.org/wiki/Graphical_user_interface">GUI</a>.</p>
<h4 id="tips">Tips:</h4>
<ul>
<li>API: Have separate unit tests for individual web services</li>
<li>Database: Create tests to add, edit or remove data</li>
<li>GUI: Focus on field-level validation</li>
<li>Use <a href="https://en.wikipedia.org/wiki/Code_coverage">branch coverage or statement coverage</a>.</li>
<li>Add some negative tests</li>
<li>As a best practice, it is more reliable to use ‘mock’ service(s) to write unit tests, in order to avoid interdependence between components</li>
<li>Think about data re-usability, and clearing any test data from the database at end of your test (if applicable)</li>
</ul>
<h4 id="tools--frameworks">Tools / Frameworks:</h4>
<p>The <a href="https://junit.org/junit5/">Junit</a> and <a href="http://testng.org/doc/">TestNG</a> unit testing frameworks are popular for a Java-based development environment. For Ruby, <a href="http://rspec.info/">RSpec</a> is widely used.</p>
<p>For the front-end, the <a href="http://karma-runner.github.io/2.0/index.html">Karma test runner</a> is commonly used in development environments using AngularJS or Vue.js to run JavaScript-based unit tests. <a href="https://www.protractortest.org/#/">Protractor</a> and the BDD-based <a href="https://jasmine.github.io/">Jasmine framework</a> are also widely used. Those using React might prefer <a href="https://facebook.github.io/jest/">Jest</a>, while Ember.js has an integrated unit testing provision. <a href="https://mochajs.org/">Mocha</a> is yet another JavaScript alternative for front-end unit testing.</p>
<h3 id="sanity-testing">Sanity Testing</h3>
<p>Sanity testing runs before any detailed functional or regression tests and tests basic, integral functionalities of an application. Sanity testing is required to run every commit to make sure critical business functionalities (such as the launching of the application) are not affected by code changes.</p>
<h3 id="system-integration-testing">System Integration Testing</h3>
<p>System Integration Testing (SIT) will touch multiple units/components and is a <a href="https://en.wikipedia.org/wiki/Black-box_testing">black box testing</a> method. Performing this testing should not require knowledge of the internal workings of the system.</p>
<p>Normally within a sprint, manual testing will be the first phase of the SIT. Later on, this will be converted to an automated test script. If there is a dedicated automation testing resource available in the Agile team, then an automation test script can be developed in parallel to development activities. The automated test script will fast-track regression testing prior to each release or every commit in the continuous integration / continuous deployment model.</p>
<h4 id="tools--frameworks-1">Tools / Frameworks:</h4>
<p>These tools and frameworks are used by the Europe PMC development team.</p>
<ul>
<li>Defect tracking tool – JIRA</li>
<li>Automation Testing tool – Selenium</li>
<li>Automation Framework – Serenity BDD</li>
<li>Programming Language – Java</li>
<li>Unit Testing Framework for Test Automation – JUnit</li>
<li>Build Tool – Maven</li>
<li>Schedule and Execution – Jenkins</li>
</ul>
<p>Automation scripts can be developed in any language, and it is not mandatory for the testing team to use the same technology that the development team uses.</p>
<h3 id="regression-testing">Regression Testing</h3>
<p>Regression testing ensures that software performs the same functionality in the same way after it is updated or changed. Regression testing should be performed after any code change to a previously developed and tested program, to ensure that new defects have not been introduced.</p>
<h3 id="user-acceptance-testing-uat">User Acceptance Testing (UAT)</h3>
<p>UAT is normally conducted by the actual users of a particular system. There will be end-to-end scenarios for the UAT users to execute to verify system functionalities. In the w<a href="https://en.wikipedia.org/wiki/Waterfall_model">aterfall model</a>, UAT will be conducted only at the end of development, whereas in the Agile model, UAT can be performed at the end of every sprint and feedback given to the team.</p>
<h2 id="common-types-of-non-functional-testing">Common types of Non-functional Testing</h2>
<h3 id="performance-testing">Performance Testing</h3>
<p>Software performance testing is used to determine the speed or effectiveness of a computer, network, software program, or device. This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions. Performance testing (“How fast is the system?”) can also include load testing (“How much volume can the system process?”).</p>
<h4 id="tools--frameworks-2">Tools / Frameworks:</h4>
<p><a href="https://gatling.io/">Gatling</a> or <a href="https://jmeter.apache.org/">JMeter</a></p>
<p><em>Sample ‘Gatling’ Performance Test Results:</em></p>
<p><a href="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST8.JPG"><img src="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST8.JPG" alt="Sample 'Gatling' Performance Test Results:" /></a></p>
<h3 id="security-testing">Security Testing</h3>
<p>Security testing is a technique to determine if an information system protects data and maintains functionality as intended. Its aims should be verifying three basic principles—confidentiality, integrity, and authentication.</p>
<h3 id="compatibility-testing">Compatibility Testing</h3>
<p>Compatibility testing involves verifying a service’s functionality is the same across various browsers, different browser versions, mobile devices, operating systems, etc.</p>
<h3 id="usability-testing">Usability Testing</h3>
<p>Usability testing is used in user-centered interaction design to evaluate a product by testing it on users. This is an irreplaceable testing practice, since it gives direct input on how real users use the system.</p>
<h2 id="automating-your-testing">Automating your Testing</h2>
<p>Test automation is the process of converting manual test scripts to automated test scripts, using testing tools like <a href="https://www.seleniumhq.org/">Selenium</a>, <a href="http://appium.io/">Appium</a>, <a href="https://www.soapui.org/">SoapUI</a>, <a href="http://rest-assured.io/">REST-Assured</a>, <a href="http://www.thucydides.info/#/">Serenity</a>, <a href="https://cucumber.io/">Cucumber</a>, etc. Test automation will fast-track regression testing, and it avoids repeated and time-consuming manual testing efforts. It is also convenient to use automated testing for compatibility testing.</p>
<p><em>Sample Automation / Cucumber-BDD-Serenity Test Report:</em></p>
<p><a href="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST7.JPG"><img src="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST7.JPG" alt="Sample Automation / Cucumber-BDD-Serenity Test Report" /></a></p>
<h3 id="benefits-of-automated-testing">Benefits of Automated Testing</h3>
<ul>
<li>Faster feedback</li>
<li>Earlier Detection of Defects</li>
<li>Accelerated Results</li>
<li>Running tests 24/7</li>
<li>Fewer human resources</li>
<li>Reusability</li>
<li>Reliability</li>
<li>Simultaneity</li>
<li>Better test coverage with multiple data combinations</li>
</ul>
<h3 id="test-flows-test-management--execution-frequency">Test Flows, Test Management & Execution Frequency</h3>
<p><a href="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST5.JPG"><img src="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST5.JPG" alt="Sample test flow" /></a></p>
<p><a href="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST6.JPG"><img src="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST6.JPG" alt="Example of execution frequency of different types of testing" /></a></p>
<!--[!['Testing in DevOps'][testing2]][testing2]
As per the **Testing Pyramid** concept displayed above, around 70% of testing efforts are at the Unit Level, approximately 20% at the Integration/Business Layer and 10% at the UI/Front-end layer.-->
<h2 id="sample-build-pipeline-model-along-with-software-testing-steps">Sample Build Pipeline model along with Software Testing steps</h2>
<p><a href="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST4.JPG"><img src="/techblog/images/posts/importance-of-software-testing-in-ci-cd/TEST4.JPG" alt="Sample Build Pipeline model along with Software Testing steps" /></a></p>
<p>In the above build pipeline model, there are three pre-production servers, i.e. Development (DEV), Testing (TEST), and Staging. Normally the DEV server will be used for all development activities, with the TEST server for functional testing by the quality assurance team, and the Staging server for performance testing and user acceptance testing. On some occasions, there will only be DEV, TEST and PROD (production) servers available. In such cases, internal testing and UAT will happen on the TEST server as well.</p>
<!--### Testing Strategy in DevOps culture [Java-based development]
[!['Testing in DevOps'][testing3]][testing3]
It is also very clear that Test Automation is not the responsibility of a DevOps engineer whereas he/she will concentrate only on the **Infrastructure Automation** rather than enhance your **codebase**-->
<h2 id="proposed-tools--framework-summary">Proposed Tools & Framework Summary</h2>
<table style="font-size:smaller">
<thead><tr>
<th>#</th><th>Description</th><th>Preferred ‑ 1</th><th>Paid?</th><th>Preferred ‑ 2</th><th>Paid?</th><th>Remarks</th>
</tr></thead>
<tbody>
<tr><td>1</td><td>Automation Framework</td><td>Serenity-BDD with Cucumber - Junit</td><td>No</td><td>Python with Behave</td><td>No</td><td></td></tr>
<tr><td>2</td><td>Front-end Automation </td><td>Selenium, Protractor</td><td>No</td><td>HP-UFT</td><td>Yes</td><td></td></tr>
<tr><td>3</td><td>Mobile Automation</td><td>Appium</td><td>No</td><td>SeeTest</td><td>Yes</td><td></td></tr>
<tr><td>4</td><td>Performance Testing</td><td>Gatling, JMeter</td><td>No</td><td>LoadRunner</td><td>Yes</td><td></td></tr>
<tr><td>5</td><td>SOAP Web Service</td><td>SOAP-UI</td><td>No</td><td>ReadyAPI</td><td>Yes</td><td>JAX-WS/JAXB concept for Test Automation</td></tr>
<tr><td>6</td><td>REST Web Service</td><td>SOAP-UI</td><td>No</td><td>Postman</td><td>No</td><td>Rest-Assured for Test Automation and Frisby with JavaScript</td></tr>
<tr><td>7</td><td>Continuous Integration</td><td>Jenkins</td><td>No</td><td>Bamboo</td><td>Yes</td><td></td></tr>
<tr><td>8</td><td>Service Virtualization</td><td>Stub-o-matic, Mockito</td><td>No</td><td>HP Service Test</td><td>Yes</td><td></td></tr>
<tr><td>9</td><td>Static Code Quality</td><td>SonarQube</td><td>No</td><td>PMD</td><td>No</td><td></td></tr>
<tr><td>10</td><td>Code Coverage</td><td>JACOCO</td><td>No</td><td>Cobertura</td><td>No</td><td>Considering the primary language is Java</td></tr>
<tr><td>11</td><td>Version Control</td><td>GitHub/GitLab</td><td>Yes</td><td>SVN</td><td>Yes</td><td></td></tr>
<tr><td>12</td><td>Defect Management</td><td>JIRA</td><td>Yes</td><td>HP ALM</td><td>Yes</td><td></td></tr>
<tr><td>13</td><td>Agile Project Management</td><td>JIRA-Agile</td><td>Yes</td><td>Trello</td><td>No</td><td>Trello free version is available with some limitations</td></tr>
<tr><td>14</td><td>Security Testing</td><td>Vega</td><td>No</td><td>ZED Attach Proxy</td><td>No</td><td></td></tr>
<tr><td>15</td><td>Penetration Testing</td><td>Metasploit</td><td>No</td><td>Wireshark</td><td>Yes</td><td></td></tr>
<tr><td>16</td><td>Build Artifacts</td><td>jFrog Artifactory</td><td>No</td><td>Nexus</td><td>No</td><td></td></tr>
<tr><td>17</td><td>CD Tool</td><td>OpenShift</td><td>Yes</td><td>Puppet</td><td>Yes</td><td></td></tr>
<tr><td>18</td><td>Cloud-based Non-Mobile execution</td><td>Amazon - AWS</td><td>Yes</td><td>MS Azure</td><td>Yes</td><td>For Test Execution and deployment</td></tr>
<tr><td>19</td><td>Cloud-based Mobile execution</td><td>SauceLabs</td><td>Yes</td><td>Amazon Device Farm</td><td>Yes</td><td></td></tr>
</tbody></table>
<h2 id="conclusion">Conclusion</h2>
<p>Considering the multiple iterations over code in the Agile/DevOps software development model, testing is one of the most important phases of software delivery. To fast-track the testing process, automated functional & non-functional test-scripts are recommended. In the ideal scenario, each Agile team may have at least one QA resource who performs the testing activities for the current sprint, as well as ensuring the preservation of existing functionalities through regression tests.</p>
<h3 id="useful-links">Useful links</h3>
<ul>
<li><a href="http://www.thucydides.info/#/">BDD Serenity</a></li>
<li>
<p><a href="https://gatling.io/">Gatling</a></p>
</li>
</ul>Rakesh NambiarSoftware testing is the process of identifying the correctness and quality of a software program. In other words, testing is executing a system or application in order to find software bugs, defects, errors or unexpected behavior. Software testing is necessary because we all make mistakes. Some of those mistakes are minor, but others can be expensive or dangerous. Especially while practicing continuous integration, continuous delivery, or continuous deployment, we need to test anything and everything we produce, because things can always go wrong. Testing is mainly classified into two types, Functional Testing and Non-functional Testing.Behavior-Driven Development in Bioinformatics2018-01-29T00:00:00+00:002018-01-29T00:00:00+00:00https://europepmc.github.io/techblog/testing/2018/01/29/behavior-driven-development-in-bioinformatics<h2 id="what-is-bdd">What is BDD?</h2>
<p><strong>Behavior-Driven Development (BDD)</strong> is a set of Software Engineering practices designed to help teams deliver more valuable and higher quality software features.</p>
<p>It adopts general techniques and principles of <a href="https://en.wikipedia.org/wiki/Test-driven_development">Test Driven Development</a> (TDD) with ideas from <a href="https://en.wikipedia.org/wiki/Domain-driven_design">Domain-driven Design</a> (DDD). BDD incrementally builds functionality guided by expected behavior.</p>
<p>A simple BDD scenario / requirement is as follows:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="nd">@TC01_EPMC_SearchTest</span>
<span class="nl">Scenario:</span> <span class="nc">Specific</span> <span class="nc">Search</span> <span class="n">by</span> <span class="nc">Keyword</span>
<span class="nc">Given</span> <span class="no">I</span> <span class="n">am</span> <span class="n">researcher</span>
<span class="nc">When</span> <span class="no">I</span> <span class="n">open</span> <span class="n">the</span> <span class="err">'</span><span class="nc">Europe</span> <span class="no">PMC</span><span class="err">'</span> <span class="nc">Website</span>
<span class="nc">And</span> <span class="nc">Enter</span> <span class="n">the</span> <span class="n">keyword</span> <span class="s">"Glycosyl transferases"</span> <span class="n">on</span> <span class="n">the</span> <span class="nc">Query</span> <span class="n">field</span>
<span class="nc">And</span> <span class="nc">Click</span> <span class="n">on</span> <span class="n">the</span> <span class="nc">Search</span> <span class="n">button</span>
<span class="nc">Then</span> <span class="no">I</span> <span class="n">should</span> <span class="n">be</span> <span class="n">able</span> <span class="n">to</span> <span class="n">see</span> <span class="n">the</span> <span class="n">matching</span> <span class="n">results</span> <span class="n">on</span> <span class="n">the</span> <span class="nc">Search</span> <span class="nc">Result</span> <span class="n">page</span></code></pre></figure>
<!--more-->
<h2 id="how-does-bdd-work">How does BDD work?</h2>
<p><a href="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd1.JPG"><img src="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd1.JPG" alt="'BDD Framework'" /></a></p>
<h2 id="benefits-of-bdd">Benefits of BDD</h2>
<ul>
<li>Understandable by both technical and non-technical people</li>
<li>Business Analysts (BA) and Subject Matter Experts (SME) can contribute BDD scenarios</li>
<li>Data can be fed along with the scenarios, which can updated any time without any changes in scripting</li>
<li>Developers can write the scripts in a language of their choice</li>
<li>Attractive test reports at the end of the execution [HTML and JSON]</li>
<li>Availability of Continuous Integration tools like Jenkins, Bamboo, etc…</li>
<li>Single platform for Web, Mobile, API, thick client, and Database Testing</li>
</ul>
<h2 id="how-bdd-fits-in-an-agiledevops-environment">How BDD fits in an Agile/DevOps environment</h2>
<p><a href="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd2.JPG"><img src="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd2.JPG" alt="'BDD Diagram'" /></a></p>
<p>After story grooming on the first day of a new sprint, the developer and test teams will agree on frozen acceptance criteria for each user story identified for the sprint. Developers should prepare BDD scenarios as per the acceptance criteria, with the scope being limited to unit-level testing. QA testers or the domain expert will write the functional scenarios for each of the acceptance criteria in the user story. Both Unit and Functional BDD scenarios may or may not contain negative scenarios. Test case design techniques such as <a href="https://en.wikipedia.org/wiki/Equivalence_partitioning">Equivalence Partitioning</a>, <a href="https://en.wikipedia.org/wiki/Boundary-value_analysis">Boundary Value Analysis</a>, and <a href="https://en.wikipedia.org/wiki/Decision_table">Decision Tables</a> can be applied while writing the BDD scenarios.</p>
<p>As a best practice, BDD functional scenarios should be sent to the BA or SME for sign-off prior to test scripting. Both development and automation test scripting activities can be carried out in parallel, based on agreement with front-end developers, on how the UI elements on the page look and what the element’s characteristics are for the script to act upon.</p>
<p>For instance, assume the agile team is going to work on a user story for the Europe PMC login screen. Both the UI/front-end developer and QA automation team must agree on a unique element locator like <code class="language-plaintext highlighter-rouge">id="login–email"</code> for the ‘Email’ field, <code class="language-plaintext highlighter-rouge">id=login–password</code> for the ‘Password’ field, etc. (see the figure, below).</p>
<p>Using this technique, the QA team will not have to wait until the development team completes the front-end and back-end development processes to start test scripting. Once the actual build is available for testing, the QA team can run the automated test script and fast track the QA tasks, detecting defects during the early stages of the sprint.</p>
<p><a href="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd3.JPG"><img src="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd3.JPG" alt="'BDD Object Identification'" /></a></p>
<h2 id="magic-behind-the-scenes">‘Magic’ behind the scenes?</h2>
<p>Developers might wonder how plain English text will trigger test execution. The answer to that question is a tool called <a href="https://cucumber.io/">Cucumber</a>, which is smart enough to parse plain text scenarios. Cucumber will either generate a function skeleton or call the respective function for each step in the given scenario (see below).</p>
<h3 id="java-steps">Java Steps:</h3>
<!-- !['BDD Java Code'][bdd4] -->
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">cucumber.api.PendingException</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">cucumber.api.java.en.Given</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">cucumber.api.java.en.When</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">cucumber.api.java.en.Then</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">cucumber.api.java.en.And</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">cucumber.api.junit.Cucumber</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.junit.runner.RunWith</span><span class="o">;</span>
<span class="nd">@RunWith</span><span class="o">(</span><span class="nc">Cucumber</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStepDefinitions</span> <span class="o">{</span>
<span class="nd">@Given</span><span class="o">(</span><span class="s">"^I am researcher$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">i_am_researcher</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">PendingException</span><span class="o">();</span>
<span class="o">}</span>
<span class="nd">@When</span><span class="o">(</span><span class="s">"^I open the 'Europe PMC' Website$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">i_open_the_europe_pmc_website</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">PendingException</span><span class="o">();</span>
<span class="o">}</span>
<span class="nd">@Then</span><span class="o">(</span><span class="s">"^I should be able to see the matching results on the Search Result page$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">i_should_be_able_to_see_the_matching_results_on_the_search_result_page</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">PendingException</span><span class="o">();</span>
<span class="o">}</span>
<span class="nd">@And</span><span class="o">(</span><span class="s">"^Enter the keyword \"([^\"]*)\" on the Query field$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">enter_the_keyword_something_on_the_query_field</span><span class="o">(</span><span class="nc">String</span> <span class="n">strArg1</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">PendingException</span><span class="o">();</span>
<span class="o">}</span>
<span class="nd">@And</span><span class="o">(</span><span class="s">"^Click on the Search button$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">click_on_the_search_button</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">PendingException</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<h3 id="javascript-steps">JavaScript Steps:</h3>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">Given</span><span class="p">(</span><span class="sr">/^I am researcher$/</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">callback</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">callback</span><span class="p">.</span><span class="nx">pending</span><span class="p">();</span>
<span class="p">});</span>
<span class="k">this</span><span class="p">.</span><span class="nx">When</span><span class="p">(</span><span class="sr">/^I open the 'Europe PMC' Website$/</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">callback</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">callback</span><span class="p">.</span><span class="nx">pending</span><span class="p">();</span>
<span class="p">});</span>
<span class="k">this</span><span class="p">.</span><span class="nx">Then</span><span class="p">(</span><span class="sr">/^I should be able to see the matching results on the Search Result page$/</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">callback</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">callback</span><span class="p">.</span><span class="nx">pending</span><span class="p">();</span>
<span class="p">});</span>
<span class="k">this</span><span class="p">.</span><span class="nx">And</span><span class="p">(</span><span class="sr">/^Enter the keyword </span><span class="se">\"([^\"]</span><span class="sr">*</span><span class="se">)\"</span><span class="sr"> on the Query field$/</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">glycosyltransferases</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">callback</span><span class="p">.</span><span class="nx">pending</span><span class="p">();</span>
<span class="p">});</span>
<span class="k">this</span><span class="p">.</span><span class="nx">And</span><span class="p">(</span><span class="sr">/^Click on the Search button$/</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">callback</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">callback</span><span class="p">.</span><span class="nx">pending</span><span class="p">();</span>
<span class="p">});</span>
<span class="p">};</span></code></pre></figure>
<h3 id="ruby-steps">Ruby Steps:</h3>
<figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">Given</span> <span class="sr">/^I am researcher$/</span> <span class="k">do</span>
<span class="c1"># do something</span>
<span class="k">end</span>
<span class="no">When</span> <span class="sr">/^I open the 'Europe PMC' Website$/</span> <span class="k">do</span>
<span class="c1"># do something</span>
<span class="k">end</span>
<span class="no">Then</span> <span class="sr">/^I should be able to see the matching results on the Search Result page$/</span> <span class="k">do</span>
<span class="c1"># do something</span>
<span class="k">end</span>
<span class="no">And</span> <span class="sr">/^Enter the keyword \"([^\"]*)\" on the Query field$/</span> <span class="k">do</span> <span class="o">|</span><span class="n">glycosyltransferases</span><span class="o">|</span>
<span class="c1"># do something</span>
<span class="k">end</span>
<span class="no">And</span> <span class="sr">/^Click on the Search button$/</span> <span class="k">do</span>
<span class="c1"># do something</span>
<span class="k">end</span></code></pre></figure>
<p>Once the function skeleton is ready, the script needs to be enhanced according to the step definition using UI automation tools like Selenium, Protractor or Appium as shown below.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="nd">@Given</span><span class="o">(</span><span class="s">"^I am researcher$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">i_am_researcher</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="n">opens_home_page</span><span class="o">();</span>
<span class="nc">ScenarioHook</span><span class="o">.</span><span class="na">getScenario</span><span class="o">().</span><span class="na">write</span><span class="o">(</span><span class="s">"Logged-In User : Scientist"</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@When</span><span class="o">(</span><span class="s">"^I open the 'Europe PMC' Website$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">i_open_the_Europe_PMC_Website</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="nc">Assert</span><span class="o">.</span><span class="na">assertTrue</span><span class="o">(</span><span class="s">"ERROR: Home Page NOT available"</span><span class="o">,</span><span class="n">homePage</span><span class="o">.</span><span class="na">verifyHomePageAvailable</span><span class="o">());</span>
<span class="o">}</span>
<span class="nd">@When</span><span class="o">(</span><span class="s">"^Enter the keyword \"([^\"]*)\" on the Query field$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">enter_the_keyword_on_the_Query_field</span><span class="o">(</span><span class="nc">String</span> <span class="n">keyWord</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="n">homePage</span><span class="o">.</span><span class="na">enterQueryString</span><span class="o">(</span><span class="n">keyWord</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@When</span><span class="o">(</span><span class="s">"^Click on the Search button$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">click_on_the_Search_button</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="n">homePage</span><span class="o">.</span><span class="na">performSearch</span><span class="o">();</span>
<span class="o">}</span>
<span class="nd">@Then</span><span class="o">(</span><span class="s">"^I should be able to see the matching results on the Search Result page$"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">i_should_be_able_to_see_the_matching_results_on_the_Search_Result_page</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Throwable</span> <span class="o">{</span>
<span class="nc">Assert</span><span class="o">.</span><span class="na">assertTrue</span><span class="o">(</span><span class="s">"ERROR: Result Result Page NOT available"</span><span class="o">,</span> <span class="n">resultPage</span><span class="o">.</span><span class="na">verifyResultsPanelAvailable</span><span class="o">());</span>
<span class="n">printMessage</span><span class="o">(</span><span class="s">"Successful"</span><span class="o">);</span>
<span class="o">}</span></code></pre></figure>
<p>In the above example, Java is used along with Selenium and jUnit.</p>
<h3 id="how-to-parameterize-the-bdd-test">How to parameterize the BDD Test</h3>
<figure class="highlight"><pre><code class="language-java" data-lang="java"> <span class="nd">@TC_01_SearchTest</span>
<span class="nc">Scenario</span> <span class="nl">Outline:</span> <span class="nc">Specific</span> <span class="nc">Search</span> <span class="n">by</span> <span class="nc">Keyword</span>
<span class="nc">Given</span> <span class="no">I</span> <span class="n">am</span> <span class="n">researcher</span>
<span class="nc">When</span> <span class="no">I</span> <span class="n">open</span> <span class="n">the</span> <span class="err">'</span><span class="nc">Europe</span> <span class="no">PMC</span><span class="err">'</span> <span class="nc">Website</span>
<span class="nc">And</span> <span class="nc">Enter</span> <span class="n">the</span> <span class="s">"<SearchKeyWord>"</span> <span class="n">on</span> <span class="n">the</span> <span class="nc">Query</span> <span class="n">field</span>
<span class="nc">And</span> <span class="nc">Click</span> <span class="n">on</span> <span class="n">the</span> <span class="nc">Search</span> <span class="n">button</span>
<span class="nc">Then</span> <span class="no">I</span> <span class="n">should</span> <span class="n">be</span> <span class="n">able</span> <span class="n">to</span> <span class="n">see</span> <span class="n">the</span> <span class="n">matching</span> <span class="n">results</span> <span class="n">on</span> <span class="n">the</span> <span class="nc">Search</span> <span class="nc">Result</span> <span class="n">page</span>
<span class="nl">Examples:</span>
<span class="o">|</span> <span class="nc">SearchKeyWord</span> <span class="o">|</span>
<span class="o">|</span> <span class="nc">Cancer</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">blood</span> <span class="o">|</span>
<span class="o">|</span> <span class="nc">Glycosyl</span> <span class="n">transferases</span> <span class="o">|</span></code></pre></figure>
<h2 id="reports">Reports</h2>
<p>By default, Cucumber generates an HTML report, and you can configure a JSON report if you wish. Cucumber-JVM is equipped to convert the JSON report displayed as below in Jenkins using the <a href="https://plugins.jenkins.io/cucumber-reports">Cucumber Reports</a> plugin.</p>
<p><a href="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd5.JPG"><img src="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd5.JPG" alt="'BDD Cucumber Report'" /></a></p>
<p>The <a href="http://www.thucydides.info/">Serenity BDD Framework</a> (a popular Java based open source framework), when used for the BDD implementation, will provide an enhanced report, as shown below.</p>
<p><a href="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd6.JPG"><img src="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd6.JPG" alt="'BDD Serenity Report1'" /></a>
<a href="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd7.JPG"><img src="/techblog/images/posts/behavior-driven-development-in-bioinformatics/bdd7.JPG" alt="'BDD Serenity Report2'" /></a>
It is also smart enough to capture a screenshot according to user settings.</p>
<h2 id="tldr">TL;DR</h2>
<p><strong>Behavior-Driven Development (BDD)</strong> is emerged from test-driven software development process that is becoming popular and widely used by Europe PMC and by many organizations, due to its transparency, and the readability of the application behaviour.</p>
<h3 id="useful-links">Useful links:</h3>
<ul>
<li><a href="https://cucumber.io/">Cucumber</a></li>
<li>
<p><a href="http://www.thucydides.info/">Serenity</a></p>
</li>
</ul>Rakesh NambiarWhat is BDD? Behavior-Driven Development (BDD) is a set of Software Engineering practices designed to help teams deliver more valuable and higher quality software features. It adopts general techniques and principles of Test Driven Development (TDD) with ideas from Domain-driven Design (DDD). BDD incrementally builds functionality guided by expected behavior. A simple BDD scenario / requirement is as follows: @TC01_EPMC_SearchTest Scenario: Specific Search by Keyword Given I am researcher When I open the 'Europe PMC' Website And Enter the keyword "Glycosyl transferases" on the Query field And Click on the Search button Then I should be able to see the matching results on the Search Result pageSwagger documentation customisation2017-11-27T00:00:00+00:002017-11-27T00:00:00+00:00https://europepmc.github.io/techblog/documentation/2017/11/27/swagger-documentation-customisation<p><a href="https://swagger.io/">Swagger</a> is a popular software framework that helps developers build RESTful Web services through their entire lifecycle, from design and documentation, to test and deployment.
This post focuses on how to incorporate the API documentation generated through Swagger inside an HTML page hosted from another web application.</p>
<p>One of the main features of Swagger is producing interactive documentation for a RESTful API. Swagger can be used in conjunction with a multitude of different languages and frameworks.
It will always produce two different outputs inside the same web application hosting the API:</p>
<ol>
<li>A default HTML page having a standard Swagger style. (<a href="https://www.ebi.ac.uk/europepmc/annotations_api/swagger-ui.html">Europe PMC Annotations API Swagger standard html page</a>)</li>
<li>A JSON file that will contain the description of the generated documentation (<a href="https://www.ebi.ac.uk/europepmc/annotations_api/v2/api-docs.json">Europe PMC Annotations API documentation descriptor</a>)</li>
</ol>
<p><!--more--></p>
<p>The goal is to display the Swagger HTML documentation inside an external web site page adapting Swagger’s default style to the one of the external site.
One option could be to copy manually the HTML produced from Swagger inside the external site page. The obvious disadvantage of this solution is that it will be necessary to keep the Swagger code contained into the RESTful API web application in sync with the content shown in the external page, each time a modification to the documentation is performed.
Ideally the content of the Swagger documentation should be displayed in the external page loading it dynamically from the RESTful API application, to avoid synchronization issues.
To this end it is necessary to perform the following steps:</p>
<ul>
<li>Download the Swagger UI folder from: <a href="https://github.com/swagger-api/swagger-ui/tree/2.x/dist">GitHub</a></li>
<li>Once downloaded, create a swagger-ui folder in the resources folder of the external web application and add the contents downloaded from the Github Repository in this folder.</li>
<li>In the header of the html page where to incorporate the Swagger documentation import the following CSS and javascript files:</li>
</ul>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><head></span>
.....
<span class="nt"><link</span> <span class="na">href=</span><span class="s">'./swagger-ui/css/typography.css'</span> <span class="na">media=</span><span class="s">'screen'</span> <span class="na">rel=</span><span class="s">'stylesheet'</span> <span class="na">type=</span><span class="s">'text/css'</span><span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">'./swagger-ui/css/reset.css'</span> <span class="na">media=</span><span class="s">'screen'</span> <span class="na">rel=</span><span class="s">'stylesheet'</span> <span class="na">type=</span><span class="s">'text/css'</span><span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">'./swagger-ui/css/screen.css'</span> <span class="na">media=</span><span class="s">'screen'</span> <span class="na">rel=</span><span class="s">'stylesheet'</span> <span class="na">type=</span><span class="s">'text/css'</span><span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">'./swagger-ui/css/reset.css'</span> <span class="na">media=</span><span class="s">'print'</span> <span class="na">rel=</span><span class="s">'stylesheet'</span> <span class="na">type=</span><span class="s">'text/css'</span><span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">'./swagger-ui/css/print.css'</span> <span class="na">media=</span><span class="s">'print'</span> <span class="na">rel=</span><span class="s">'stylesheet'</span> <span class="na">type=</span><span class="s">'text/css'</span><span class="nt">/></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/object-assign-pollyfill.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/jquery-1.8.0.min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/jquery.slideto.min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/jquery.wiggle.min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/jquery.ba-bbq.min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/handlebars-4.0.5.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/lodash.min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/backbone-min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/swagger-ui.min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/highlight.9.1.0.pack.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/highlight.9.1.0.pack_extended.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/jsoneditor.min.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/marked.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">'./swagger-ui/lib/swagger-oauth.js'</span> <span class="na">type=</span><span class="s">'text/javascript'</span><span class="nt">></script></span>
<span class="nt"></head></span>
.....</code></pre></figure>
<ul>
<li>In the body of the same html page it is necessary to place a div where the Swagger documentation will be placed and then call the Swagger specific function to load the relevant API documentation</li>
</ul>
<figure class="highlight"><pre><code class="language-html" data-lang="html"> <span class="nt"><body></span>
.....
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"swagger-ui-container"</span><span class="nt">></div></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span><span class="nt">></span>
<span class="nx">jQuery</span><span class="p">(</span><span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">hljs</span><span class="p">.</span><span class="nx">configure</span><span class="p">({</span>
<span class="na">highlightSizeThreshold</span><span class="p">:</span> <span class="mi">5000</span>
<span class="p">});</span>
<span class="c1">// Pre load translate...
</span>
<span class="k">if</span><span class="p">(</span><span class="nb">window</span><span class="p">.</span><span class="nx">SwaggerTranslator</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">SwaggerTranslator</span><span class="p">.</span><span class="nx">translate</span><span class="p">();</span>
<span class="p">}</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">swaggerUi</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">SwaggerUi</span><span class="p">({</span>
<span class="na">url</span><span class="p">:</span> <span class="dl">"</span><span class="s2">https://www.ebi.ac.uk/europepmc/annotations_api/v2/api-docs.json</span><span class="dl">"</span><span class="p">,</span>
<span class="na">dom_id</span><span class="p">:</span> <span class="dl">"</span><span class="s2">swagger-ui-container</span><span class="dl">"</span><span class="p">,</span>
<span class="na">supportedSubmitMethods</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">get</span><span class="dl">'</span><span class="p">],</span>
<span class="na">docExpansion</span><span class="p">:</span> <span class="dl">"</span><span class="s2">list</span><span class="dl">"</span><span class="p">,</span>
<span class="na">jsonEditor</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">defaultModelRendering</span><span class="p">:</span> <span class="dl">'</span><span class="s1">example</span><span class="dl">'</span><span class="p">,</span>
<span class="na">showRequestHeaders</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">showOperationIds</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">validatorUrl</span><span class="p">:</span><span class="kc">null</span>
<span class="p">});</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">swaggerUi</span><span class="p">.</span><span class="nx">load</span><span class="p">();</span>
<span class="p">});</span>
<span class="nt"></script></span>
...
<span class="nt"></body></span></code></pre></figure>
<p>The javascript object SwaggerUi will load the API documentation described through the url property and will place it inside the dom identified by the dom_id property value.
Below are details of the properties that can be set to the SwaggerUi object:</p>
<table>
<thead>
<tr>
<th>Parameter Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>url</td>
<td>The url pointing to <code>swagger.json</code> (Swagger 2.0) or the resource listing (earlier versions) as per <a href="https://github.com/OAI/OpenAPI-Specification/">OpenAPI Spec</a>.</td>
</tr>
<tr>
<td>authorizations</td>
<td>An authorization object to be passed to swagger-js. Setting it here will trigger inclusion of any authorization or custom signing logic when fetching the swagger description file. Note the object structure should be <code>{ key: AuthorizationObject }</code></td>
</tr>
<tr>
<td>spec</td>
<td>A JSON object describing the OpenAPI Specification. When used, the <code>url</code> parameter will not be parsed. This is useful for testing manually-generated specifications without hosting them. Works for Swagger 2.0 specs only.</td>
</tr>
<tr>
<td>validatorUrl</td>
<td>By default, Swagger-UI attempts to validate specs against swagger.io's online validator. You can use this parameter to set a different validator URL, for example for locally deployed validators (<a href="https://github.com/swagger-api/validator-badge">Validator Badge</a>). Setting it to <code>null</code> will disable validation. This parameter is relevant for Swagger 2.0 specs only.</td>
</tr>
<tr>
<td>dom_id</td>
<td>The id of a dom element inside which SwaggerUi will put the user interface for swagger.</td>
</tr>
<tr>
<td>booleanValues</td>
<td>SwaggerUI renders boolean data types as a dropdown. By default it provides a 'true' and 'false' string as the possible choices. You can use this parameter to change the values in dropdown to be something else, for example 0 and 1 by setting booleanValues to new Array(0, 1).</td>
</tr>
<tr>
<td>docExpansion</td>
<td>Controls how the API listing is displayed. It can be set to 'none' (default), 'list' (shows operations for each resource), or 'full' (fully expanded: shows operations and their details).</td>
</tr>
<tr>
<td>apisSorter</td>
<td>Apply a sort to the API/tags list. It can be 'alpha' (sort by name) or a function (see Array.prototype.sort() to know how sort function works). Default is the order returned by the server unchanged.</td>
</tr>
<tr>
<td>operationsSorter</td>
<td>Apply a sort to the operation list of each API. It can be 'alpha' (sort by paths alphanumerically), 'method' (sort by HTTP method) or a function (see Array.prototype.sort() to know how sort function works). Default is the order returned by the server unchanged.</td>
</tr>
<tr>
<td>defaultModelRendering</td>
<td>Controls how models are shown when the API is first rendered. (The user can always switch the rendering for a given model by clicking the 'Model' and 'Model Schema' links.) It can be set to 'model' or 'schema', and the default is 'schema'.</td>
</tr>
<tr>
<td>onComplete</td>
<td>This is a callback function parameter which can be passed to be notified of when SwaggerUI has completed rendering successfully.</td>
</tr>
<tr>
<td>onFailure</td>
<td>This is a callback function parameter which can be passed to be notified of when SwaggerUI encountered a failure was unable to render.</td>
</tr>
<tr>
<td>highlightSizeThreshold</td>
<td>Any size response below this threshold will be highlighted syntactically, attempting to highlight large responses can lead to browser hangs, not including a threshold will default to highlight all returned responses.</td>
</tr>
<tr>
<td>supportedSubmitMethods</td>
<td>An array of of the HTTP operations that will have the 'Try it out!' option. An empty array disables all operations. This does not filter the operations from the display.</td>
</tr>
<tr>
<td>oauth2RedirectUrl</td>
<td>OAuth redirect URL</td>
</tr>
<tr>
<td>showRequestHeaders</td>
<td>Whether or not to show the headers that were sent when making a request via the 'Try it out!' option. Defaults to <code>false</code>.</td>
</tr>
<tr>
<td>jsonEditor</td>
<td>Enables a graphical view for editing complex bodies. Defaults to <code>false</code>.</td>
</tr>
</tbody>
</table>
<ul>
<li>Now it is possible to customize the standard style of the Swagger documentation tweaking both the default Swagger css and the site application css as required.
An example of the result can be found at <a href="https://europepmc.org/AnnotationsApi">Europe PMC Annotations API documentation page</a></li>
</ul>Francesco Talo'Swagger is a popular software framework that helps developers build RESTful Web services through their entire lifecycle, from design and documentation, to test and deployment. This post focuses on how to incorporate the API documentation generated through Swagger inside an HTML page hosted from another web application. One of the main features of Swagger is producing interactive documentation for a RESTful API. Swagger can be used in conjunction with a multitude of different languages and frameworks. It will always produce two different outputs inside the same web application hosting the API: A default HTML page having a standard Swagger style. (Europe PMC Annotations API Swagger standard html page) A JSON file that will contain the description of the generated documentation (Europe PMC Annotations API documentation descriptor)