Upgrading InterMine
See our blog for details on each of the InterMine releases. You can view the release notes and associated tickets on GitHub too.
Upgrade Instructions#
For non-disruptive releases, you can upgrade your mine by incrementing your version number in your mine's gradle.properties file:
Read more: InterMine Versioning Policy and InterMine Development Roadmap``
InterMine 5.0.7#
This is a non-disruptive release.
It contains a new generic TSV, CSV file loader and a few new web services.
InterMine 5.0.6#
This is a non-disruptive release.
It contains log4j upgrade.
InterMine 5.0.5#
This is a non-disruptive release.
It contains log4j upgrade.
InterMine 5.0.4#
This is a non-disruptive release.
It mainly contains new webservices to improve BlueGenes support (v. 1.3.0).
InterMine 5.0.3#
This is a non-disruptive release.
It contains updated interpro and interpro-go properties files and a fix to the Enrichment widget.
InterMine 5.0.2#
This is a non-disruptive release.
InterMine 5.0.0#
This release contains:
- new web services to support BluGenes 1.0.0,
- implementation of the Frictionless Data Package specs,
- migration from jcenter to maven central,
- deletion of deprecated Bioschemas DataRecord profile.
info
To be able to download intermine artifacts, you need to apply some changes on the build.gradle of your MINE in order to replace jcenter() repo with mavenCentral(). See biotestmine build.gradle for an example.
This release still supports the legacy (JSP based) user interface but we strongly encourage to use BlueGenes instead.
Configure BlueGenes#
- Add any addition credit displayed on the bottom of the home page, configuring the project.credit web properties.
- Configure the WHAT'S NEW section on the home page, adding the project.news web property.
- Configure the CITE US section on the home page, updating the project.citation property with the publication URL.
- Configure any existing external oauth2 provider (as Google or Elixir) adding an authorised redirect URIs pointing to BlueGenes /api/auth/oauth2callback endpoint. More instructions here.
- Update the blueGenesStart task defined in your MINE/webapp/build.gradle file, adding the new environment variable
bluegenes-default-namespace
See biotestmine for an example. We recommend NOT to use blueGenesStart task in production.
Firefox cache#
note
If you use Firefox, you might experience an issue due to the browser cache, in that case you need to delete the cache. To delete the cache go to Preferences -> Privacy & Security -> Clear Data and then check the box Cached Web Content.
InterMine 4.3.1#
This is a non-disruptive release.
InterMine 4.3.0#
This is a non-disruptive release.
InterMine 4.2.0#
This is a non-disruptive release.
It mainly contains new webservices to improve BlueGenes support (v. 0.10.0), some fixes related to the type Date, and bioschemas markup on the report page (Gene and protein)
InterMine 4.1.3#
This is a non-disruptive release.
It contains a small batch of bug fixes.
InterMine 4.1.2#
This is a non-disruptive release.
InterMine 4.1.1#
This is a non-disruptive release.
It contains some bug fixes related to ncbi-gff bio source and few improvements from ThaleMine.
You can build your mine using Gradle wrapper 4.9. To update the version, run the following command in your InterMine instance directory:
See our blog post for more details (https://intermineorg.wordpress.com/2019/10/29/intermine-4-1-1-patch-release/).
InterMine 4.1.0#
This is a non-disruptive release.
Galaxy integration has been improved; you should remove the galaxy related properties from the web.properties file to benefit from it.
Integration with ELIXIR AAI has been included.
Gradle wrapper updated to the 4.9 version.
Some bug fixes.
See our blog post for more details (https://intermineorg.wordpress.com/2019/09/24/intermine-4-1-0/)
InterMine 4.0.1#
Restore Strains to core data model.
InterMine 4.0.0#
DataSet.licence was added to the data model. To update to this new data model for this release, you'll want to rebuild your database and redeploy your webapp.
To enable the structured data added to the web pages in format of JSON-LD, you should set the property markup.webpages.enable to true in the web.properties file.
To configure the new URLs used in the "share" button, you should specify the keys in the class_keys.properties file.
See our blog post for more details on how to configure and use the new features to make your mine more FAIR.
InterMine 3.1.2#
This is a non-disruptive release.
InterMine 3.1.1#
This is a non-disruptive release.
InterMine 3.1.0#
The class Strain was added to the core InterMine data model in this release.
You will need to rebuild your database with the new model to release
a new webapp.
If you do have Strains in your data, you might think about using the
core data classes now available.
To update to use the new InterMine release:
Change your mine's
gradle.propertiesfile to3.1.+.Change your data sources'
gradle.propertiesfile to3.1.+.
InterMine 3.0.0#
This release adds Solr to InterMine. To upgrade, you will need to rebuild your database and install Solr.
To Upgrade#
Change your mine's
gradle.propertiesfile to3.0.+. If you have data sources, change the version they use too.Install Solr
Configure Solr
Rebuild your database.
Specifically the postprocesses that build the search index.
You should then be able to deploy your webapp as normal, with the new and improved search.
InterMine 2.+#
InterMine 2.0 is a disruptive release and is not backwards compatible. This means that databases, webapps and code from previous releases will need to be updated to work with the new InterMine release. Below are detailed instructions on how to migrate your InterMine to the new build system.
Warning If you have custom InterMine code, your changes will likely not work as expected after the upgrade. Please contact us and we can help you migrate your edits to the new system.
Please contact us if you have any questions or concerns! We have a mailing list or you can contact us directly via email or our discord channel (chat.intermine.org). If you are having difficulties, we can also arrange a skype call to walk through any problems together. Please make sure your code is public, e.g. GitHub, so we can help test!
Gradle#
InterMine now uses Gradle to manage dependencies and to build and run InterMine. Please see Gradle Quick Start for useful Gradle commands and Gradle FAQs for help with common questions and errors.
See the Gradle blog post for details as to why we made this change.
Maven#
You will need Maven installed. We use Maven to manage mine-specific InterMine dependencies, including your mine-specific data parsers.
You do not need to install Gradle locally. Instead, use the Gradle wrapper provided.
Remove InterMine code#
Previously, you had to download and compile InterMine. Now, instead, you'll be using the compiled InterMine JARs available via Maven. This means you should remove all InterMine code from your mine repositories. Your mine repositories should only contain your mine (webapp and dbmodel) and your mine's custom data sources.
If you have your mine and bio/sources in your InterMine checkout, instead of in their own repository, you'll have to separate them out.
What you want to end up with:
FlyMine - https://github.com/intermine/flymine/ (MUST be the name of your mine)
FlyMine specific data sources -
Options to separate out your mine repo:
You can copy over your directories directly. However, don't do this! You'll lose your history.
Instead, use
git filter-branchcommand. Follow the directions on how to move a directory to a new repository and keep your history in GitHub.
You should not have any core InterMine code locally.
New directory structure#
InterMine now uses the standard Maven directory structure.
You will have to run two migration scripts to move your current mine over to this new layout -- one script for your mine and one for your mine's data parsers. The migration scripts are located in the intermine-scripts repository.
Migrate mine webapp to new directory structure#
Run "migrateMine" script to move your mine over to the new directory system. You might want to create a new gradle branch for testing.
Migrate data sources to new directory structure#
Run the "migrateBioSources" script to move your sources over to the new directory system.
Run this command to put your sources on the classpath and therefore, available to the database build:
This task builds the JARs and places them on your classpath in ~/.m2/repository.
Note the command is ./gradlew instead of gradle. Use the provided Gradle wrapper instead of locally installed Gradle.
You will have to install your sources every time you update the source code in order to update the JAR being used by the build.
Previously, the data model was merged from all data sources' additions XML file. This is no longer true. Since each source is in its own JAR now, the data model is self-contained. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source:
Update config#
Remove
<property name="source.location" location="../bio/sources/"/>from your project XML file.Set
GRADLE_OPTSinstead ofANT_OPTS- Use the same parameters.
- Append
-Dorg.gradle.daemon=falseto prevent daemons from being used.
Update project XML for some sources:
SOsource's location has been updated to be:<property name="src.data.file" location="so.obo"/>Protein2iprsource has a new attribute:<property name="osAlias" value="os.production"/>intermine-items-xml-fileisn't a valid value for "type" anymore. Use the project name instead.src.data.dircan only have alocationattribute.src.data.dircannot have a
valueattribute.Change the location of the generated files for
entrez-organismandupdate-publicationsdata sources to beorganisms.xmlandpublications.xml(instead of in thebuilddirectory)
InterPro data file needs to be updated. The file incorrectly references
interpro.dtdwhen you should have the full path instead.Update interpro.xml
<!DOCTYPE interprodb SYSTEM "ftp://ftp.ebi.ac.uk/pub/databases/interpro/interpro.dtd">I asked InterPro to fix but they said no. Maybe you could ask too?
See https://github.com/intermine/intermine/issues/1914 for the
discussion.
Update each data source's additions file to be correct. Alternatively you can use the
extraAdditionsFile(see previous section).PostprocessUtil.javamoved to thebiopackage, so you maybe have to update your import to beimport org.intermine.bio.util.PostProcessUtil;.
Please see Gradle Quick Start for details on Gradle and common Gradle commands and Gradle FAQs for help with common questions and errors.
Data Model#
Syntenic Regions have been added to the data model.
Protein.molecularWeight is now a Float instead of an Integer.
GO evidence codes now have a name and URL.
OntologyAnnotation can now annotate any InterMine object, as long as
that class inherits
Annotatable.Sequence Ontolgy has been updated to the latest version.
Organism.taxonId is a String instead of an Integer.
See the Model Changes blog post for details.
You may have to update your data sources and queries to match the new data model.
Dependencies#
Software dependency requirements have been updated to the latest versions. This is so that we can get rid of legacy code and make use of new features.
You will get errors if you use older versions. e.g. If you use Java 7, you will get this error: Caused by: java.security.NoSuchProviderException: no such provider: SunEC
API changes#
We are making some non-backwards compatible changes to our API. These three end points have a parameter called xml which holds the XML query. We are going to rename this parameter to be query (as we now accept JSON queries!) to match the syntax of all the other end points.
Please update any code that references these end points.
Pre-InterMine 2.0 Upgrade Instructions#
To pull changes in your local repository and merge them into your working files, use this command:
If you host a copy of the CDN, then you should also pull in changes from that repository.
Upgrade to InterMine 1.6#
The core model of InterMine has changed in release 1.1, so you may encounter more errors than usual.
update integration keys
You may need to update your integration keys if they are using a class or field that’s been changed.
update custom converter
If you are storing data using a class or field that’s been changed, you will have to change your code to use the new model. See the complete list of model changes below.
template queries
You will have to update your templates to use the new model.
interaction viewer
The cytoscape tool uses the new model. It will not work until you build a database with the new code
Interactions
| class | old | new |
|---|---|---|
| Interaction | gene1 | participant1 |
| gene2 | participant2 | |
| relationshipType (Term) | relationshipType (String) | |
| InteractionDetail | allInteractors (Gene) | allInteractors (Interactor) |
| Interactor | – | stoichiometry |
| InteractionDetail.role1 | role | |
| InteractionDetail.type | type |
Protein Domains
| class | old | new |
|---|---|---|
| ProteinDomain | proteins | proteinDomainRegions |
| Protein | proteinDomains | proteinDomainRegions |
| ProteinDomainRegion | – | start |
| – | end | |
| – | identifier | |
| – | database |
Upgrade to InterMine 1.4#
There are no model changes, but we've added some new features that require an update.
We've added a fancy new connection pool for performance improvement. However, you do need to update some configuration files.
Postgres config file#
The number of database connections required will depend on your usage. 100 connections is the default and should be okay for production webapps. However, each webapp reserves 20 connections. So, on your dev machines, it may be wise to raise the maximum quite a bit.
{% tabs %} {% tab title="postgresql.conf" %} max_connections=250 {% endtab %} {% endtabs %}
$MINE properties files#
in your $MINE directory:
{% tabs %} {% tab title="default.intermine.integrate.properties" %} set
db.production.datasource.maxConnections=20
db.common-tgt-items.datasource.maxConnections=5
and for each database, replace
db.production.datasource.class=org.postgresql.ds.PGPoolingDataSource
(or any other db pooling class)
with these 2 lines
db.production.datasource.class=com.zaxxer.hikari.HikariDataSource db.production.datasource.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource
{% endtab %}
{% endtabs %}
{% tabs %} {% tab title="default.intermine.webapp.properties" %} set
db.production.datasource.maxConnections=20
and for each database, replace
db.production.datasource.class=org.postgresql.ds.PGPoolingDataSource
(or any other db pooling class)
with these 2 lines
db.production.datasource.class=com.zaxxer.hikari.HikariDataSource db.production.datasource.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource
{% endtab %}
{% endtabs %}
Any other data source you use should be set to five connections and increased to ten if you encounter problems, e.g. the build failing with an error like so:
{% tabs %} {% tab title="Error message" %} Caused by: org.postgresql.util.PSQLException: FATAL: connection limit exceeded for non-superusers {% endtab %} {% endtabs %}
Or this (See #912)
{% tabs %} {% tab title="Error message" %} Unable to get sub-ObjectStore for Translating ObjectStore {% endtab %} {% endtabs %}
See HikariCP and InterMine settings for details.
InterMine-model Refactor#
The metadata package has moved to InterMine-model. If you have custom data sources that use InterMine Utils, you may have to update your code to reflect the new location. Your IDE should be able to do this for you.
Tomcat#
Add clearReferencesStopTimerThreads to your $TOMCAT/conf/context.xml file, so it should look like so:
Upgrade to InterMine 1.3.x#
This code will work with any webapp and database created with InterMine 1.3+.
Upgrade to InterMine 1.3#
Removed all duplicate entries from web.xml.
Model changes include the following:
DataSet now has a publication reference.
AnnotationExtension has been moved from GOAnnotation to
GOEvidence.
Also, we have changed our GO parser a bit. Each line in a gene annotation file now corresponds with an Evidence object. In prior releases, each Evidence object was unique e.g. only a single evidence code per gene / GO term pair.
Upgrade to InterMine 1.2.1#
If you have your own home page (begin.jsp), you must manually make this change: 501e221
This is a fix for the keyword search - when users submit a blank search form, see Issue #329
There are no model or configuration changes in this release.
Upgrade to InterMine 1.2#
The core data model has not been changed, so you should be able to release a webapp using InterMine 1.2 code without making any changes.
Upgrade to InterMine 1.1#
The core model of InterMine has changed in release 1.1, so you may encounter more errors than usual.
update integration keys
You may need to update your integration keys if they are using a class or field that’s been changed.
update custom converter
If you are storing data using a class or field that’s been changed, you will have to change your code to use the new model. See the complete list of model changes below.
template queries
You will have to update your templates to use the new model.
interaction viewer
Widget uses the new model - will not work until you build a database with the new code.
Model Changes#
Updated to latest version of Sequence Ontology, 2.5
| old | new |
|---|---|
| Comment.text | Comment.description |
| Gene.ncbiGeneNumber | – |
| – | Gene.description |
| – | Gene.briefDescription |
Interactions#
| class | old | new |
|---|---|---|
| Interaction | gene | gene1 |
| interactingGenes | gene2 | |
| type | details.type | |
| role | details.role1 | |
| – | details.role2 | |
| name | details.name | |
| shortName | – | |
| InteractionRegion | primaryIdentifier | – |
| name | – |
Gene Ontology#
| class | old | new |
|---|---|---|
| GOAnnotation | withText | evidence.withText |
| with | evidence.with | |
| – | annotationExtension | |
| OntologyTerm | – | crossReferences [1] |
[1] used for Uberon
Identifiers#
We have several [wiki:Homologue new homologue data converters] available in this InterMine release. However, some of these new data sources use Ensembl IDs. If you want to load the model organism database identifier instead (important for interoperation with other InterMines), you should use the Entrez Gene ID resolver. To do this:
Download the identifier file -
Unzip the file
Add the path to properties file like so:
Configuration Updates#
Web services use the webapp.baseurl property to run queries, so be sure this is the valid URL for your mine. Otherwise, you will get an "Unable to construct query" error on the query results page.