extensions

Tripal Devel

Tripal Devel

1. Recently funded projects for Tripal development.

   1. NSF DIBBS:

      1. http://www.nsf.gov/news/news_summ.jsp?cntn_id=132880

      2. http://nsf.gov/awardsearch/showAward?AWD_ID=1443040&HistoricalAwards=false

      3. API for cross-communication and web services.

      4. Integrate Galaxy with Tripal.

      5. Software defined networking.  

   2. NRSP: Breeding resources, Customizable query building, open-source applications for field collections of breeding data and linking with Tripal.

   3. NSF PGRP

2. Tripal Workshop at PAG: Call for Extension Modules to be presented

   1. Existing modules

      1. ND Natural Diversity Genotypes (U Sask)

      2. Blast Tripal Modules  (U Sask/Univ Iowa/WSU)

      3. Tripal Germplasm  (U Sask)

      4. Functional Annotation Modules (KEGG/InterPro)

      5. JBrowse module (U Sask)

      6. Tripal Daemon module (U Sask) (forgot to mention in call, added afterwards)

   2. What’s coming

      1. EImage Module (WSU)

      2. QTL Module (Iowa / WSU)  (See demo later today at PeanutBase under QTL tab).

      3. Main lab Customizations for Genotypes/Searching (WSU).

      4. Network Module / Expression data (Discuss with Mara / Kris)

   3. Modules included in other talks: Andrew’s Phylotree module (NCGR) in Ethy’s talk.

3. Module Development:  Drupal "git vetted user role"

   1. Please email mailing list to have other users review your modules.  We can review each other’s code.

   2. To place your module on Drupal.org and to have permission to release full projects follow the steps on this page: https://www.drupal.org/node/1011196

4. Other ongoing projects

   1. GenSAS: automated and manual structural and functional annotation of genomic sequence. Integrates with WebApollo.  Probably release Q1 of 2015.  With Tripal integration coming later. (WSU)

   2. WebApollo Tripal module (needs lots of work)  (WSU)

   3. JBrowse Tripal module for integrating JBrowse with Tripal (Univ Sask)

5. What else is needed?

   1. Nelson:  Issues of access permissions for specific data types.

      1. Valentin: uses chado controller, works with feature table primarily, and allows you to restrict access to specific records.  Thinking about creating a Tripal module for install for chado controller.

      2. CHADO Controller URL: http://www.gnpannot.org/content/chado-controller

      3. Added Issue to track issue: https://www.drupal.org/node/2352859

6. Feature requests.

   1. Mara:  More command-line ways to interface with Tripal.  Specifically because she has pipelines to interact with Chado database. (i.e. creating organism/feature pages, etc). Issue added at: https://www.drupal.org/node/2314235

   2. Mara:  Blast loader requires a database cross-reference.  Can we handle blasting without specifying a database.  Issue added at https://www.drupal.org/node/2352329

   3. JBrowse integration example: http://coffee-genome.org/jbrowse

7. Chado

   1. Scott: Updated release perhaps by end of 2014.

   2. Mara: Group module, new update with Flybase requests.  

8. TODO:

   1. Next meeting have a shared google doc.  

Next Month’s Agenda

1. Inclusion of new extension modules as part of Tripal (if they are generic enough)

2. Should Tripal be broken into separate modules?

1. Reminders

   1. Module Development:  Drupal "git vetted user role"

   2. Please email mailing list to have other users review your modules.  We can review each other’s code.

   3. To place your module on Drupal.org and to have permission to release full projects follow the steps on this page: https://www.drupal.org/node/1011196

2. Funded Projects:

   1. NSF DIBBs (PI Ficklin)

      1. Tripal Exchange Module:  REST web services.  Work begins now. First developer’s meeting held mid December. All invited. Watch for forthcoming email.

         1. Andrew farmer implemented API for gene family module (based on phylotree module).

         2. Ethy has had requests for web services for QTL module.

      2. Tripal Galaxy Module:  Work will begin later in 2015.

      3. Tripal SDN Module:  exploration of SDN technology for faster data  transfers between Tripal databases.

   2. USDA NRSP (PI Main). Formal list of items coming next month.  Started in October.

   3. USDA SCRI (PI Main). Formal list of items coming next month.  Started in October.

      1. NCBI parser/loader/Gene/Gene family module (year 1)

      2. Tripal module for RightField (done in year 1)

      3. Tripal Module for Field Book (breeding data application for collecting phenotype data (pictures, measurements)) (year 2).  Any relation to IBP field book?  Will be completely open-source.

      4. Customizable Query builder tool (year 1 & 2)

      5. Conversion of Breeders Toolbox to be Tripal compatible.

         1. breeding data management (year 1-3)

         2. other breeding decision tools (year 2-5)

   4. Andrew, Ethy, Steve,... NSF PGRP proposal outstanding.  Legume database federation.

   5. Nobel Foundation: Alfalfa. Extension of Chris Town Medicago database.

3. Job Postings:

   1. Post on Tripal Site (with expiration date) if desired.

   2. Post on GMOD Site (coordinate with Scott).

   3. Current Openings:

      1. Web Developer at WSU for DIBBs project.

4. Upcoming Activities:

   1. Posted on Tripal Website. Let us know and we will add them.

   2. Tripal PAG Workshop. Jan 11th, Sunday at 4:00-6:15pm PST.

      1. Extension Modules presented by Lacey:

         1. ND Natural Diversity Genotypes (U Sask)

         2. Blast Tripal Modules  (U Sask / Univ Iowa / WSU)

         3. Tripal Germplasm  (U Sask)

         4. JBrowse module (U Sask)

         5. Tripal Daemon module (U Sask)

         6. EImage module (if finished in time)

         7. Phlyotree module ??

         8. QTL module ??

   3. GMOD PAG Workshop.  Jan 14th, Wed 10:30-5:00pm PST.

   4. GMOD Online Training / Summer School ?

   5. GMOD Annual Meeting ?

   6. Others ??

      1. BOSC:  Dublin Ireland. 2 days before ISMB conference. Provide support for hackathons.

   7. DIBBs web services design meetings (Start mid Dec 2014)

   8. Any conferences where your work will be presented?

   9. Continued Tripal Developer Meetings (change to every other month rather than monthly?)

      1. Interleave… some more topic and every other month the larger broader meeting.

5. Extension Modules Under Development

   1. QTL Module (Iowa / WSU / USask)  

   2. Main lab Customizations (WSU).

   3. Network Module / Expression data (WSU / Clemson; Discuss with Mara & Kris).

   4. GenSAS: automated and manual structural and functional annotation of genomic sequence. Itegrates with WebApollo.  Probably release Q1 of 2015.  With Tripal integration coming later (WSU).

   5. WebApollo Tripal module (needs lots of work)  (WSU).

   6. VCF Loader (WSU).

   7. Pedigree/Relationship Viewer (USask).

      1. Does anyone need tree-like views of relationship data.

      2. Ontologies are configurable via the interface.

      3. Using JavaScript InfoViz Toolkit (JIT).

      4. Check the phylotree module from Andrew’s group.  Visualization based on D3 libraries.

   8. ND Genotypes (has a Released version; documentation needed) (USask)

   9. Tripal Germplasm & Cross Management (USask)

   10. Genbank Parser. Includes search tool. Will implement in group module. Includes bulk loader templates for importing Genbank files (WSU)

   11. Phylotree Module (NCGR/Iowa)

   12. Shopping Cart style data store (NCGR). Generating lists of items to do operations.  

       1. Light-weight list generating tool in Tripal and passing those/integrating with iPlant.

       2. Uses LightShop Drupal module.

       3. Implemented for BioMaterial module already.

6. Tripal Core Future Developments

   1. What is core functionality?  Should web services be incorporated as core functionality?  What data loaders should come with Tripal by default (i.e. VCF, GenBank)? Should only Tripal Core module be Tripal and all other modules be extensions?   

      1. Reason for the question:

         1. Too much stuff, is it overwhelming?

         2. Takes a long time to convert to new Drupal version & Release updates.

         3. Currently, Tripal is heavy on the whole genome / transcriptomics which may confuse folks who may be looking to construct a resource for storing stocks/germplasm, breeding resources, etc.

         4. Maintenance: hard to keep up with all these modules on the same release schedule.

      2. Pros for separating

         1. Allow for more releases, more often

         2. Simplified code base to convert as new versions of Drupal are released.

         3. Users only need download modules they need. They don’t get a bunch of other modules they don’t need.

         4. Forces dependencies to be properly specified (currently all come together).

      3. Cons:

         1. Increases the difficulty for installation:  users’ must be more aware of dependencies and which modules to install.

      4. Concerns:

         1. The core modules must remain generic for common usage.  By making other modules not “core” (e.g. feature) there is the risk of losing oversight and loss of generic functionality.  Who maintains?

      5. * TODO:  plan out what should go into core, what should be separate (specific to a certain subgroup).

   2. RSS feed on Tripal.info that lists extensions available.  

      1. Add to the Tripal core module a web interface to parse read that RSS feed and show users all available modules.

      2. Tag modules with categories to help select modules for installation.

   3. Installation Profiles:  

      1. RSS feed provides profile setup details.

      2. pre-configured set of modules by need (e.g. genomics site, breeding site, genetics site, etc…).

      3. functions just like a manually installed Tripal site with the ability to add additional modules, disable core ones, etc.

   4. Data access permissions.  

      1. Proposed expansion of web services to include authentication/access NSF PGRP (PI Main @ WSU)

      2. Current solution:  chado controller

   5. Expanded Drush commands for full command-line management -- no current funding to do this.  Must be done step-by-step.

   6. Tripal changes for Drupal 8

      1. Use of Drupal Entities (object oriented)

         1. Provides an API for every chado table.

   7. Store results of queries into a “Shopping Cart” style.

7. Chado

   1. Updated release

      1. List of new tables

   2. * Progress getting into git.

   3. Group Module

8. iPlant Data Commons (Steve Cannon)

   1. A new feature at iPlant.  Website development toolkit within iPlant.  Tools for sharing managing, publishing data to be used with iplant analysis tools.  Connects to workflows metadata, etc.

   2. Looks like a new GMOD-style toolkit.  For creating custom websites within iPlant.

   3. * Talk with iPlant during development to have design interaction.

   4. Perhaps PAG Meeting to discuss.

9. Issues

   1. Tripal GFF3 loader doesn’t create polypeptide features.

10. Topics for future meeting

    1. Open up discussion of germplasm / cross management / breeding.

    2. Javascript graphing visualization library.

Gerard: Status of Chado/Tripal.

• Not sure of Chado status need to ask Scott

• Tripal:  exploration of future development for object oriented and restructuring of how modules are delivered and support of other data storage mechanisms.

• Development of Web Services

• Integration with Galaxy.

Nathan: ETA for 2.0 stable release.  Tripal doesn’t yet have unit testing.

Gerard: Security Issue for D7 sites.  

Solutions:  

• For sites without active addition of content: have a staging site for checking before production.  Have a scheduled release for updates.  

• For sites with active addition of content:  not sure what do ….  Need a solution,

• Stay on top of updates.  Install as quick as they come out.

Ethy/Steve:  Kick off meeting to construct Tripal QTL meeting at PAG @12pm.  

• To help with conformity between: Ames sites, WSU sites, U. Sask.

• Blast module:  will bring info for that to PAG as well.

Stephen F:  Will Phylotree module be release soon.

Sunday evening dinner.  Walking distance.  New fresh food restaurant in the mall towards the far end of the mall.

Action Item

• Ask Scott about current status of Chado.

• Invite to web services discussion at PAG.

• Get an ETA for a 2.0 expected release.

• How to restore sites from a potential hack event if content is actively added.

1. Thanks to PAG speakers!

2. Status of Chado

   1. Ideas for changes: changes to organism table for infraspecific names; Group module needs some looking at further; adding property tables for existing tables (also how to deal with units);  Eimage table; db table in it’s own module. And some changes requested on the mailing list.

   2. Will make document of changes public for other comments.

   3. Idea: deal with easy issues (organism table, property tables) for a quick release and more difficult issues in future release.

   4. Move to GitHub… request for any further commits.

3. GMOD Summer School (Tripal workshop) + GMOD Meeting

4. GMOD/Google Summer of code

   1. “Google Summer of Code is a global program that offers student developers stipends to write code for various open source software projects”

   2. http://gmod.org/wiki/GSoC

5. Status of Tripal v2.0 stable release.  Look for release in March.

6. Tripal extension modules currently under development

   1. QTL  (U. Iowa, WSU, U. Sask)

      1. Need to pull together a vocabulary and then a list of requirements for feedback from other groups.

      2. Waiting on feedback of template file.

   2. BLAST (U. Iowa, U. Sask)

      1. plan to put development code in GitHub and move to Drupal repository once it’s ready for general use.  

      2. Separate but somehow combined to reduce confusion.

   3. Networks (WSU, Clemson)

   4. Web Services (WSU + community)

      1. Demo currently online, using AGAVE + HAL format.

      2. Need to address data privacy issues within the web services.

         1. differential access (different groups have access to differenet things).

      3. Anonymous registration to track usage.

   5. Phylogeny (NCGR, U. Iowa)

      1. Make public soon.

   6. Pedigrees (U Sask)

   7. d3 support (U Sask)

      1. want to support basic pie/bar charts, feature location diagrams, etc.

   8. KEGG module issues (WSU)

   9. Ecommerce style shopping cart (NCGR) for collecting data. Currently gene oriented has some sequence download functionality.

7. Searching?

   1. Add a section on how to deal with searching to the online documentation?

   2. Valentin Guignon provided a suggestion for integration with ElasticSearch and Solr and provided some example code: https://www.drupal.org/node/2214775

   3. Need a single search box capability (google type approach).  Perhaps something like Entrez.

8. Setup Improvements?

   1. Installation, ease of use, recommended improvements?

   2. Installation profiles vs Distribution

      1. Installation profiles can be executed after Drupal is setup to download everything needed.

      2. Distribution is a full package with everything need that can contain an installation profile to complete some additional steps.

   3. What kind of “Tripal” Distributions/Installation profiles should we offer?

      1. Genomics site for whole genomes transcriptomes?

      2. Breeding-specific site?

      3. E-commerce/stock site?

      4. Need tutorials for these.

      5. Other thoughts

         1. Not include default organisms, analyses. Or ability to control what data gets preloaded.  

         2. Ethy offers to provide feedback from her group.

         3. Include a tour in distributions: https://www.drupal.org/project/joyride

         4.  

9. Change to gene/mRNA default page:  merge into a single gene page?

   1. difficult to traverse feature relationships (mostly gene centric).

   2. Yes, combine, at least for genes, into a single gene page.

   3. Chris Childers: i5K would be willing to share their efforts as a potential default solution.

10. TODO:  

    1. Add a section to the Tripal tutorial about permissions… how to make some date private for certain groups.

• VCF “non” loader.

  • How does this compare to how we would store this data anyway. Can an indexed

• Associating files with data.

  • Use Cases:

    • BAM files for RNA-Seq / integrated with Expression Modules

    • VCF

    • Whole genomes: FASTA, GFF, Excel.

• Multi-Chado and Chado Controller for Tripal: some updates (about the projects) for the community (by VG).

  • View initial design: https://drive.google.com/open?id=1q4rWcncmqvDseV3Np6NldoYKFIikXMhNqvXAUQEzbTc&authuser=0

  • Chado Controller

    • Sandbox: https://www.drupal.org/sandbox/guignonv/2428743

    • Enforces constraints at the PostgreSQL level (not Drupal)

      • Can then use artemis/apollo or other software that has direct connection to the database (including Tripal).

      • Works by using PLSQL and views/triggers/rules/procedures.

    • Chado history: extension of Chado Audit module.

      • Records a history of edits/updates/inserts for a table in a mirror table

      • differences with the audit module: more audit columns to include user and transaction identifiers

    • Access restrictions: limits read/write access at a granular level.

      • Works on feature table at the moment but will support other tables soon.

      • ‘feature’ table is renamed to ‘feature_data’, and a ‘feature’ view takes the place of the feature table. PostgreSQL user account are not allowed to query ‘feature_data’ table but can query the ‘feature view’.

    • Annotation Inspector

      • Allows for review of annotations made by reviewers.

      • Allows to automate some changes according to the data that has been modified by somebody. (ex. auto-change a color property according to a gene annotation state).

      • Still some redesign to be done and it will be renamed into “Chado Inspector”.

      • Use cases:

        • could be run by an annotator on his favorite genes to check his work.

        • could be run on the whole database by and admin to have statistics and see what’s good and not so good.

        • automate tasks.

        • force some changes regardless what the users asked; for instance the “owner” property of a gene so the annotator who changed the gene can’t use a different name that his user name (ie. sometimes “John Smith”, “JOHN”, “jsmith”, whatever... only his real user name will be used regardless what he entered as a value for the “owner” property).

    • Performance?

      • Audit module has no noticeable impact.

      • Annotation inspector is not optimized, and cause slowness depending on the number of rules and elements on which the rules are checked.

      • Access restrictions, not fully explored in the context of Tripal usage. May have noticeable impact. Impact might only be at the first connection time.

  • Available as an “alpha” Drupal Sandbox module.  

    • Currently only for annotation history: https://www.drupal.org/sandbox/guignonv/2428743

  • Multi-Chado access (separate from Chado Controller)

    • Sandbox: https://www.drupal.org/sandbox/guignonv/2429515

    • Needs updates to core module to support this.

    • Access restriction will support multiple chado instances.

    • Use Cases:

      • public and a private version of a same chado database in 2 different schema, “chado” and “chado_private”; the Tripal site is public; anonymous people will only see what’s in “chado” schema; users that are logged on will see what’s inside “chado_private”;

      • Different funding sources necessitate separating database.

    • Possible use for access control for web services as long as the access can be limited through PostgreSQL access layer (GRANT/REVOKE) at PostgreSQL user level (ie. “REVOKE ALL ON TABLE feature FROM ROLE some_user;”).

• Improvements / Changes to standard modules.

  • Instead of a TOC have all data in a single page

  • Publication module:  Need viewers for all linking tables.

  • Stock images attach to stock pages and link to GRIN.

    • Possible EImage image coming soon

  • Need to accommodate genetic maps created from crosses of multiple species (some may be synthetic).  

    • multi-species and synthetic crosses in stock table

    • required organism field in stock table points to a genus organism record

    • individuals in stock table are linked to multiple organisms (perhaps via a new stock_organism linker table)

    • featuremap records are linked to stocks via a custom linker table, featuremap_stock.

    • features (e.g. markers, linkage groups) in the genetic map are assigned to the genus organism record since the organism field must be set in the feature table.

    • need perhaps an accommodation in Chado to support this.

    • Additional use case: interspecific crosses (ie: Lens culinaris by Lens ervoides) cannot currently be stored in chado since you have to pick a single organism for each stock.

• Stable 2.0 version still planned for March.

• Next meeting Apr. 7: a formal agenda meeting.

• Status of Tripal stable release v2.0 -- May 31st.

• Changes to tripal.info site.

• Future Updates for Tripal

  • Entities for Drupal 8.

  • Installation profiles.

    • Make links for these notes online tripal.info.

    • Lacey has a generic installation profile. It’s close to be usable already. Lacey will reinvigorate efforts on this.  She’ll explore including this in stable release.

  • Changes to default pages.

  • Addition of a Tripal module repository resource on tripal.info to make finding extensions easier.

    • Add a list of modules being worked on to help reduce duplication of effort.

    • Folks should also list new modules on listserv

    • Should add a document for how to share modules on tripal.info.

  • Integration of Andrew Farmer’s (NCGR) Phylonode module.

  • Unit Testing (to speed testing for release).

    • Steve Cannon may provide a student to help with development of unit testing.

    • Stephen Ficklin will add a section to the Developers Handbook for how to implement unit testing.

  • Documentation Updates

    • Add instructions for setting permissions on page level

    • Add instructions for integrating non-Drupal Search

    • Is the documentation useful?  How to improve?

• Tripal default page interface

  • Ethy, Meg: don’t like the collapsible sidebar.

  • Ethy suggested wrapping panels under the sidebar if space is available.

  • Ethy likes the breadcrumb trail.  But how can we design this.

  • Steve mentioned the circular pages to gene/mRNA pages.  

    • On gene pages give this information for the primary transcript by default.

    • Chris: i5K folks will be willing to share styles and templates.  

    • Need to add a place to share templates on tripal.info an extension theme.

  • Need a way to more easily enhance templates on a “type” basis (for features, stocks, etc...)

  • Need to add a missing templates for pub linker (and other linker).

• Any new work? Questions?

• Status of Extension Modules

  • QTL  (U. Iowa, WSU, U. Sask)

    • Sook and Ethy are trying to get it started up again.  Nice to have more involvement.  Need someone to implement it as there isn’t a much time perhaps in the summer.  Goal is to have a working draft by PAG 2016.

    • Perhaps expand into genetic module to include markers and maps.

    • Meg expressed interest in being involved.

    • Lacey expressed interest in keeping genetic markers separate from QTLs.

  • BLAST (U. Iowa, U. Sask)

    • http://peanutbase.org/blast & http://legumeinfo.org/blast

    • Code: https://www.drupal.org/project/tripal_blast

    • New feature: hit visualization, but in need of a quick repair to image of ‘target’ sequence.

    • Ames development effort has a github instance which makes pull requests to a master development github instance. Stable versions of the code is pushed to the Drupal repository.

  • Networks (WSU, Clemson)

    • https://www.drupal.org/sandbox/spficklin/1796198

  • Web Services (WSU + community)

    • Part of Tripal 7.x-2.x on branch 7.x-2.x-toc

  • Phylotree (NCGR, U. Iowa)

    • https://github.com/ncgr/tripal_phylogeny

    • http://legumeinfo.org/search/phylotree

  • Pedigrees/d3 support (U Sask)

    • https://www.drupal.org/project/tripald3

    • There is a current production release at the URL above

    • Future plans include moving pedigrees out to a separate module (perhaps in the same download) and further developing a Tripal D3 API for developers allowing for consistent themeing, pop-ups, etc. between modules.

  • KEGG module issues (WSU)

    • https://www.drupal.org/project/tripal_analysis_kegg

  • Chado Controller (Bioversity International)

    • https://www.drupal.org/sandbox/guignonv/2428743

  • Multi-Chado (Bioversity International)

    • https://www.drupal.org/sandbox/guignonv/2429515

  • Ecommerce style shopping cart (NCGR) for collecting data. Currently gene oriented has some sequence download functionality.

    •  

  • File module (associating images too). (WSU)

    • https://www.drupal.org/sandbox/spficklin/2458713

  • Daemon API/Drush Daemon API / Tripal Daemon (USask)

    • https://www.drupal.org/project/drushd

    • https://www.drupal.org/project/tripal_daemon

    • Current releases available at the above URLs

    • Development is on hold pending suggestions from users on functionality they would like to see

  • Others?

• Next Meeting May 5th, an informal agenda.

1. Meeting notes are now archived on the Tripal.info site:  http://tripal.info/meetings/developers/summary

2. Strategies for dealing with large numbers of features: memory and slowness issues. (EC)

   1. Related to Tripal issue https://www.drupal.org/node/2463211.

   2. Problematic pages:

      1. feature admin.

         1. Timeout in the first instance has been traced to very slow queries to count, retrieve, and sort all features.

         2. FIX:

            1. change the select boxes to text boxes in the view.

            2. disable the sorting will speed up the query.

            3. loose index scan is faster than a select DISTINCT. See notes posted by Nathan below for solution to code.

            4. The pagination does a query to get the count and second to populate the page.

            5. We can setup the view so we only display results when the user presses the submit/filter button

      2. organisms with many features

         1. Feature browser: not very useful (Ethy).  Maybe have some sort of example page to replace the feature page.  Perhaps a block.

         2. Feature summary.

      3. maps with many features.

         1. possible solution would be to limit the nested objects that are returned from chado_generate_var().

   3. Problematic processes:

      1. Sync-ing (it is possible to go directly to sync page, which by-passes the time-out on the feature list pages).

      2. Updating URLs (though much improved now that it no longer runs out of memory).

         1. Want to select to a subset of features (by type, organism, or simply a count).

      3. Removing orphaned features.

         1. Runs out of memory.

   4. Partition feature table some way? Or the queries? Alternative methods for counting records?

      1. Suggest using a bigserial for primary keys in Chado.

      2. Need to explore table partitioning in Chado.  Valentin will be potentially exploring this.  

   5. Multiple schema? (E.g. one per organism) But how to deal with foreign keys across schema? Could feature be replaced with a materialized view?

      1. For questions please contact Valentin for how this could be done.

   6. Statistics for the database:  Taein Lee is working on a module to show the number and amount of data types.  To provide information about structure in the database in relation to classes.

   7. Sync-ing a large number of features/stocks in a bulk fashion.

      1. memory problems on syncing stocks.

      2. drush command. **

      3. You can limit the number of stocks you want to sync at one time

3. Search Engine for Chado: https://www.drupal.org/node/2214775

   1. feedback on the use of elasticsearch search API module (by Valentin)
      
      http://www.crop-diversity.org/mgis/accession-search

      1. The ElasticSearch Drupal module ( https://www.drupal.org/project/search_api_elasticsearch) is new and under development.

      2. More details: https://www.drupal.org/node/2214775#comment-9895567

   2. has anyone worked on incorporating ontologies into searches for tripal? (ADF).

      1. A project is in the works at Ames to take advantage of ontology hierarchy to search for data objects attached to ontology terms. Stephen is doing something similar with REST services. Will collaborate.

   3. Add to the Tripal User’s Guide a section on options for setup of Searching.

   4. USDA/ARS is working on integrating Solr into the Tripal site to index Drupal/Chado and other file types.  

      1. Two modules: Drupal Solr (well supported) and an additional module to add file support (not well supported)

4. Mailing lists (gmod-tripal vs gmod-tripal-devel).

5. Updates to Extension page on tripal.info.  Forthcoming interface on Tripal sites for easier installation.

6. Issues we didn’t cover, but will be on agenda for next month’s meeting:

   1. Downloads: it would be nice to have downloads built in to Tripal, especially downloads of search results, configurable as to which fields from the record pages to include. In the meantime, what have other groups done? (EC) NB: this relates to an issue I raised for our site regarding the views-data-export module listed as a "Highly Recommended Module" in the README that comes with tripal (ADF)

   2. Permissions: having to manually reset permissions for modules on install seems to trip us up frequently during development/deployment. Is there a reason that modules should not have some default permission settings and if not, is there a way to effect this? (ADF)

Action Items

1. Stephen will fix the slowness on the admin pages and memory issues for stable release v2.0

2. Stephen will experiment with bigserial for pkeys in Chado.

Loose Index Scan Notes:

It turns out that the primary bottleneck in the 3rd query is the "SELECT DISTINCT":

SELECT distinct(type_id) FROM chado.feature

 

This is apparently implemented by doing a sequential scan on the chado.feature table:

drupal=> explain SELECT distinct(type_id) FROM chado.feature;

                              QUERY PLAN                               

------------------------------------------------------------------------

HashAggregate  (cost=176753.74..176753.85 rows=11 width=4)

  ->  Seq Scan on feature  (cost=0.00..165521.79 rows=4492779 width=4)

 

I stumbled upon the following optimization, called a "loose indexscan", which works well when there are many rows containing few distinct values:

https://wiki.postgresql.org/wiki/Loose_indexscan

Using that to implement a faster equivalent to the 3rd query results in a run time of ~5 milliseconds (vs over 13 _seconds_):

LOG:  statement: WITH RECURSIVE t AS (

         SELECT MIN(type_id) AS type_id FROM chado.feature

         UNION ALL

         SELECT (SELECT MIN(type_id) FROM chado.feature WHERE type_id > t.type_id)

         FROM t WHERE t.type_id IS NOT NULL

       )

       SELECT cvterm_id, name FROM chado.cvterm WHERE cvterm_id IN

         (SELECT type_id FROM t WHERE type_id IS NOT NULL) ORDER BY cvterm.name ASC;

LOG:  duration: 4.762 ms

 

I think we should update tripal_views_handler_filter_select_cvterm.inc to use the new query.

--- a/tripal_views/views/handlers/tripal_views_handler_filter_select_cvterm.inc

+++ b/tripal_views/views/handlers/tripal_views_handler_filter_select_cvterm.inc

@@ -67,7 +67,7 @@ class tripal_views_handler_filter_select_cvterm extends tripal_views_handler_fil

        $where = ' AND ' . implode(' AND ', $where_clauses);

      }



-      $sql = "SELECT cvterm_id, name FROM {cvterm} WHERE cvterm_id IN (SELECT distinct(" . $this->field . ") FROM {" . $this->table . "}) " . $where . ' ORDER BY cvterm.name ASC';

+      $sql = "WITH RECURSIVE t AS (SELECT MIN(" . $this->field . ") AS col FROM {" . $this->table . "} " . ($where == '' ? '' : "WHERE " . $where) . " UNION ALL SELECT (SELECT MIN(" . $this->field . ") FROM {" . $this->table . "} WHERE " . $this->field . " > col " . $where . ") FROM t WHERE col IS NOT NULL) SELECT cvterm_id, name FROM {cvterm} WHERE cvterm_id IN (SELECT col FROM t where col IS NOT NULL) ORDER BY cvterm.name ASC";

      $resource = chado_query($sql);

      $cvterms = array();

• Tripal v2.0 stable release:  released June 1st!

  • We should have  more frequent releases

  • Changes between v2.0-rc1 and v2.0:

    • http://www.tripal.info/news/tripal_v2.0

• Overview/Progress of changes since last meeting

  • Admin pages are faster.

    • Forms don’t perform search (show data) unless submitted

    • Drop-downs populated with loose index scan (patch supplied by Nathan).

    • Feature admin page has d3 bar graph summary of features.

      • We want user input on this feature before adding it to all of the admin pages.

      • Lacey will explore making this bar graph available for anonymous viewers.

  • URL setting will have a ‘type’ filter setting but it is not yet there.

  • Memory leaks have been fixed for removing orphaned features and syncing features..

  • Feature browser is now deprecated in v2.0, but is still available in the event someone is using it.  It will go away in some release in the future.

  • Add a drush command for syncing:  This didn’t make it in the v2.0 release but an issue was added. https://www.drupal.org/node/2498995

  • Chado v1.3 should have bigserial for the pkey fields to help with large tables.

    • There should be a Chado v1.3 release soon!

    • Tripal will be Chado v1.3 compatible shortly thereafter. A new Tripal release will be made at that point.

    • Ask Scott about the type_id on the organism table.

    • Andrew:  how to handle features associated at higher taxonomic ranks than genus.

    • Steve & Ethy: how to handle features at the genus level (or synthetic crosses). How to link featuremaps (currently using stocks for these). For featuremaps linked to stock record, and then linked the stock record to multiple stocks and those secondary stocks are linked to features. Create an organism record to represent multiple species.

      • Sook suggested:  markers are assigned to the organism from which the source DNA was extracted.  Then the marker feature can be associated with any maps even if organism doesn’t match.

      • Lacey:  for features that are for various species use ‘spp’ in species field of organism table.

• Add / update your extensions on the tripal.info site.  http://tripal.info now supports logins.

• Upcoming Changes / Features / Events

  • Phylogeny module release.

  • Other modules with upcoming releases?

  • Contemplating migration of Tripal to GitHub for improved code sharing and forking

    • A synced copy of the code will continue to remain on Drupal.org

    • Drupal.org will continue to provide releases.

    • Possibility of splitting the core Tripal modules into separate packages for easier code sharing (e.g. phylogeny module).

  • Web services design meeting.

• Needs?

  • Valentin:  Is there a way to remove (or replace) data that was added via the bulk loader. For instance, for the stock_cvterm table. I have a stock linked to the cvterm ‘in vitro collection’ (through stock_cvterm table) and I’d like to have it linked (during an update job) to the cvterm ‘cryo-preserved’ instead. It means either removing previous relationship and adding the new one or updating the stock_cvterm.cvterm_id. Those 2 terms belong to the same CV but other terms from that CV might be used for current stock and should not be removed.

  • Andrew:  Need a public accessible way to view terms with information about parental terms (tree to show relationships).

  • Sook/Jing:  How to add bulk load publication that are not available in PubMed or Agricola.   How about a bibtex loader?  Lacey may make a bulk loader template. Will add an issue to the Tripal issue queue.  

    • Steve Cannon:  EndNote exports to flat files and a perl script is used to load those pubs.  Will be happy to share that script.  Can post on Tripal.info site:  1) Documentation for how to dump from EndNote and convert to flat file.  2) Use perl loader to import.

    • Valentin:  how can we integrate with the Drupal biblio module.

• Items from last meeting agenda that were not discussed

  • Downloads: it would be nice to have downloads built in to Tripal, especially downloads of search results, configurable as to which fields from the record pages to include. In the meantime, what have other groups done? (EC) NB: this relates to an issue I raised for our site regarding the views-data-export module listed as a "Highly Recommended Module" in the README that comes with tripal (ADF)

    • Lacey uses the Drupal module views_data_export.  But she’s not very happy. It can be very slow and doesn’t clean up interrupted downloads.

      • Lacey is willing to share cleanup script for cleaning files.

      • Ex. of use: http://www.crop-diversity.org/mgis/bulk-export

        • Incorporates views_dynamic_fields project on Drupal.org. https://www.drupal.org/project/views_dynamic_fields

      • Chun-huai has a custom search module with a download function which runs plain SQL and uses PHP to generate CSV file for users to download.

    • Stephen: will add an issue to the Tripal issue queue.  Update the documentation to indicate the issues with the views_data_export.

  • Permissions: having to manually reset permissions for modules on install seems to trip us up frequently during development/deployment. Is there a reason that modules should not have some default permission settings and if not, is there a way to effect this? (ADF)

    • Lacey: setting permission by default may make security assumptions not desired by others.

      • Should we set ‘view’ permissions by default for authenticated users?

    • Chun-huai:  add a global setting to set default module permissions.

      • Stephen: will add an issue to explore this option.

    • Andrew other site-specific settings… a topic to discuss for more use cases….  Stephen: add an issue so we don’t forget it.

• Any other items?

  • Sudhansu: Help/user-guide feature for modules (e.g., Phylotree module). Need a way to provide help for users.  Is working on a module(a generic include file for now) for adding ‘Help’ tabs to each content type page.  The inc file creates a small block at the top of a target page with a short message and an optional link to a more detailed help page. The detailed help page is included as an html file in the module dir. So, the code is not dispersed and stays with the module for portability.

    • Lacey: Drupal working towards interactive site tours.  She is creating this for knowpulse.  The tools is a JavaScript library called Shepherd.  Another javascript library is Joyride but Lacey feels it wasn’t as accurate as highlighting the exact element she wanted.

    • http://github.hubspot.com/shepherd/docs/welcome/ shows an example (obviously color is configurable :) )

    • Stephen:  Can use collapsible fieldset that contains the help text.

    • Maybe meet later in July - August for a separate meeting on help.

• Next meeting July 7th.

• Action Items:

  • Get input on admin charts (email sent to mailing list, poll on tripal.info)

  • URL/title setting needs a type filter: https://www.drupal.org/node/2470789

  • Follow-up with Scott:

    • organism.type_id constraint

    • how to handle features associated at higher taxanomic rank than genus

  • Make space available on tripal.info for sharing of Steve Cannon et al.’s perl script for loading publications

  • Update documentation stating Views Data Export as “Highly Recommended”

    • https://www.drupal.org/node/2499297

  • Lacey: Share drush script for cleaning up cancelled views data export tables (Mentioned in https://www.drupal.org/node/2499297 )

  • Add Issues:

    • Feature Request (Valentin): allow bulk loader to delete records

      • https://www.drupal.org/node/2499299

    • Feature Request (Andrew): public cvterm description pages (showing hierarchy would be helpful)

      • https://www.drupal.org/node/2499301

    • Feature Request (Sook/Jing, Lacey): method to bulk load publications (look into developing a bibtext loader, explore integration with Drupal Biblio module)

      • https://www.drupal.org/node/2499303

    • Feature Request: Bulk Data Export. Document problems with views data export.

      • https://www.drupal.org/node/2499297

    • Feature Request: setting default permissions for Tripal; explore the addition of a global setting making it easier to set these permissions.

      • https://www.drupal.org/node/2499305

    • Task: Come up with a best practices for module help & user guide. Collaborate with Sudhansu & Lacey.

      • https://www.drupal.org/node/2499309

    • Feature Request (Andrew): More site-specific settings on module pages

      • https://www.drupal.org/node/2499311

    • Feature Request (Andrew): Be able to display new feature admin chart to users.

https://www.drupal.org/node/2499881