Tripal: uploading NCBI gff

[kyrenya]

Hi all,
I'm trying to add genomic data following the steps of the citrus tutorial without success:

my gff file is downloaded from NCBI and its entries look like this:
(first 20 lines)

When I try to upload the gff following the steps (no advance options added), I Get this error message:

I've tried several options: adding first the FASTA, adding first the GFF, adding to the advance options in the gff uploader the Landmark Type field, but I cant find the correct way of doing it.

Anyone could help me?
Thanks a lot in advance

[spficklin]

Hi @kyrenya. The GFF loader is complaining because it cannot find the NW_019805504.1 feature in the database. The easiest way to get those into the database is to load a FASTA file with those sequence names. You mention you tried loading a FASTA file. What does that FASTA file contain? Can you post some of the header lines from the FASTA file you are trying to load?

[kyrenya]

Thank you so much for your super-fast answer! Sure! I forgot!! it's in fact the most stranger thing to me, because the fasta file looks like this:

[spficklin]

Okay so that looks good. And to confirm you had no errors after importing that FASTA file? And just to verify... are you using Tripal v2 or Tripal v3?

[kyrenya]

No errors after importing the fasta and I'm using v3 :)

[spficklin]

Are the files you are uploading via FASTA/GFF publicly available so that we can try loading to duplicate the problem?

[kyrenya]

Yes! Sorry, I could have include all this info at once.. I'm a noobie :)

https://www.ncbi.nlm.nih.gov/genome/?term=quercus+suber

[kyrenya]

I'm using GCF_002906115.1_CorkOak1.0_genomic.fna and
GCF_002906115.1_CorkOak1.0_genomic.gff files.

[kyrenya]

Due to the big size of files, I used shortened versions of these files.

Also, I only managed to make it work, after changing the gff file content:

(original structure from .gff):

EDITED structure:

this way, ID matches with the SEQNAME.

Is there a straight-forward way of uploading this gff via the tripal options, avoiding to edit the gff original from NCBI?

I still didnt try to upload the big files, but I'll confirm it works when I do it.

Thanks!

[spficklin]

@kyrenya it took me a while to find time to look into this. I fully loaded the entire FASTA file named GCF_002906115.1_CorkOak1.0_genomic.fna and it loaded fine in about 10 minutes. I then started loading the entire GFF file named GCF_002906115.1_CorkOak1.0_genomic.gff and I am not able to reproduce the problem you describe above. It seems to be progressing just fine. The only warnings I get are these:

Cannot find unique landmark feature: '<em class="placeholder">XM_024050139.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024050139.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024050144.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024050144.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024050144.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024050144.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024050144.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024049968.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024049968.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024049968.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024049968.1</em>'.
Cannot find unique landmark feature: '<em class="placeholder">XM_024049968.1</em>'.

But those are related the rows in the GFF of type cDNA_match which have Targets that are not in the database, so that is to be expected.

So, I am not able to reproduce the problem. You indicated you used shortened version of the files. Can you verify that in your shortened version of the FASTA file that all of the landmark sequences are present for the same ones in the shortened GFF file?

[kyrenya]

Hi again! If I understood well the question "an you verify that in your shortened version of the FASTA file that all of the landmark sequences are present for the same ones in the shortened GFF file", the answer is yes. I saved in the three files (gff, fasta and rna fasta) all the entries selected to keep in the shortened version. And it run ok, as you described behind, but only once I edited the gff giving the ID the same name as the SEQNAME.
Maybe I should have loaded the genomic fasta first and then the original NCBI gff file? I uploaded first the gff in order to folllow the tutorial steps.

Thanks a lot for your help.

[kyrenya]

Hi again,
I'm running v3 with ubuntu 18.04.
Last time I was trying to upload gff, fasta, etc, I was using shortened files to learn the process.
I am currently using a complete gff file (downloaded from ncbi), and its now 5 days since it is running the job... is that normal?

Thanks!

[kyrenya]

13 days uploading gff file.. and it still didnt finish.. any help or comments?

[bradfordcondon]

hi, no it isnt. I'll download the data and check.

Is there any information in your logs? Did you run via drush or cron? if via drush can you share the terminal output?

[kyrenya]

well, it didnt finish.. so basically what I've been seeing the last 13 days is this:

the numbers keep changing.. so I think its actually scanning the gff...

[bradfordcondon]

You dont have features in your database named "XM_02404400.1" etc so it will fail and be slow.

BEFORE you load the GFF, you must load a FASTA file with the landmark featurse. In this case, thats the genomic fasta file on the project page you linked: ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/002/906/115/GCF_002906115.1_CorkOak1.0/GCF_002906115.1_CorkOak1.0_genomic.fna.gz

Load those features first using the FASTA loader, then rerun the GFF importer.

[kyrenya]

OK, I will try and let you know!
Thanks!

[bradfordcondon]

i see this is exactly what stephen suggested above.

Can you share the output of the FASTA importer with us so we can verify it runs without error?

edit:

grep 'XM_02404400.1' GCF_002906115.1_CorkOak1.0_genomic.fna
________________________________________________________________________________
| bc:~/Downloads$
| =>

as stephen noted the genomic file is just missing missing these features.

Can you contact the people who created this dataset and ask about the missing features? It's going to be slow because of the error logging, but theres no way around it...

oops: actually i need to pay more attention. its the file is full of NW tagged features, and we are looking for XM tagged features. We must need to load a different FASTA file prior to loading the GFF.

edit: also, XM_02404400.1 isnt a real term and isnt in the GFF file. .... i must be crosseyed trying to type it out from the screenshot, so lets just ignroe it for now.

[bradfordcondon]

OK here's my advice:

verify you have loaded the genomic scaffold file. then, ALSO load the transcript FASTA file: ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/002/906/115/GCF_002906115.1_CorkOak1.0/GCF_002906115.1_CorkOak1.0_rna.fna.gz

Since these transcripts are getting reffed in a way that the GFF loader wants them pre-loaded.

If there are still transcripts giving you an error that it cant find, paste them here.

[kyrenya]

Oops, true story, sorry for that, at some point, I re-started this thing and looked again in how I did with the short files that worked fine following the docs order, I didnt thought ion the impact of the warnings for a huge file.

[bradfordcondon]

typically if you are getting that "cant find landmark error" it means you forgot to load the reference FASTA file first and you should cancel the job, load the FASTA, and try again :)

[kyrenya]

ok, just to be sure:

I'm chosing (loading the FASTA GCF_XX_genomic.fna)
sequencetype: region
method: update only
Namematch type UNIQUE NAME

Reg Exp for Unique name ^(\S*)

loading the GCF_XXX_rna.fna:

seqtype: mRNA
method update only
namematch: name

the mRNA fasta entries look like:

lcl|NW_019805508.1_mrna_XM_024058643.1_2 [gene=LOC112022923] [db_xref=GeneID:112022923] [p
roduct=putative leucine-rich repeat receptor-like serine/threonine-protein kinase At2g19230
] [transcript_id=XM_024058643.1] [location=join(32..119,220..409,1306..1746)] [gbkey=mRNA]

I'm assuming I have to make a regexp for detecting the XM_##### inside [transcript_id=XM_#####]
correct?
would the Regexp [t.*(XM_\d+) make the work?

thanks thanks!

[bradfordcondon]

Reg Exp for Unique name ^(\S*)

i think this is your problem. Have you tried not providing a regexp?

>NW_019805504.1 Quercus suber isolate HL8 unplaced genomic scaffold, CorkOak1.0 scaffold_1, whole genome shotgun sequence

by default (no regexp), this sequence will get loaded in as NW_019805504.1 which is what you want.

Please ignore the mRNA for now, and reload the genomic regions fasta with no regexp.

method: update only

THis is also incorrect, you need to set it to insert. Update means it wont insert new features, which is what you want to do!

As I said, please paste the output of trying to load the genomic fasta so we can confirm it worked.

[kyrenya]

ok! its running! I'll paste the output as soon its done!

[kyrenya]

done!

[kyrenya]

method: update only

THis is also incorrect, you need to set it to insert. Update means it wont insert new features, which is what you want to do!

true. Yes, Update Only was ok when I uploaded the gff before the fasta.. not the case now! thx for noticing!

[kyrenya]

Hi again! Shall I upload the mrna file now? for easier reading.. I repeat here my doubts concerning the upload of htis file:

loading the GCF_XXX_rna_from_genomic.fna:

seqtype: mRNA
method: update only
namematch: name

the mRNA fasta entries look like:

lcl|NW_019805508.1_mrna_XM_024058643.1_2 [gene=LOC112022923] [db_xref=GeneID:112022923] [p
roduct=putative leucine-rich repeat receptor-like serine/threonine-protein kinase At2g19230
] [transcript_id=XM_024058643.1] [location=join(32..119,220..409,1306..1746)] [gbkey=mRNA]

I'm assuming I have to make a regexp for detecting the XM_##### inside [transcript_id=XM_#####]
correct?
would the Regexp [t.*(XM_\d+) make the work?

thanks thanks!

[bradfordcondon]

im still confused why we would need to load the mRNA to get the GFF to load. I'm goign to try loading just the genomic and then the GFF myself and see if i get errors.

[bradfordcondon]

After running myself, it looks like the GFF 99% loads no problem assuming you just load the genomic features fasta without a regexp.

You get errors for perhaps 10 landmark features. You get these errors because those features were aligned to this target feature. IE, it references a transcript that hasnt been loaded yet.

If you want the warning to go away you will need to load the mRNA features such that the uniquename is XM_024067939.1

Your regexp according to my checking it would miss the .1 at the end

how about t.*(XM_\d+\.\d+)

It does appear that the GFF loader "hangs". I've never seen this before, and im hoping its because of the transcript issue.

https://www.ncbi.nlm.nih.gov/nuccore/XM_024067939.1

If we can just download the right FASTA file, you cuold load without regexp.

Here is the fasta downloaded at teh above link:

>XM_024067939.1 PREDICTED: Quercus suber DEAD-box ATP-dependent RNA helicase 20-like (LOC112035065), mRNA
ATGAGAGTCAAGAGCGTTGAAATTGTAATTGCGACACCTGGGAGGTTGATTGATATGTTGGATTCGCATC
ATATGAACTTGCGAAGGGTTCCTTATCTTGTGTTGGATGAGGCAGATCGGATGTTAGATATGGGCTTTGA
GCCTCAGATTCGAAAGCTTGTTTCTCAGATTCGCCCAGATTGTCAAACTTTGTACTGGAGTGCTACCTGG
CCTAAAGAGGTTGAACAACTTGCTAGACAGTTTCTTTACAACCCGTACAAAGTGATAATAGGATCTCAAG
ATTTGAAAGCTAACCATGCTATACGCCAACACGTTGACATTGTTTCTGAAAACCAGAAATATAACAAATT
GGTTAATTTGCTGGAGGATATCATGGATGGCAGCCAAATATTGATCTTCATGGATACCAAGAAAGGATGT
GATCAGATTACCTGGCAGCTTCGCATGGATGGCTGGCCAGCTCTTTCAATTCATGGATATAAAAGTTAAG
CAGAGAGGAACTGGTCCTCAGTCCTCTCAGAGTTTAAAGCTGGCAAGAGTCCTGTAATGACTGCTACGGA
TGTTGCAGCTTGTGGTTTAGATGTGAAGGATGTGAAATATGTGATAAATTATGACTTCCCTGGATCCCTT
GAGGACTATGTTCACCGTATTGGTCAAACAGGAAGGGCCGGGCCAAAAGGAACTGCATACACTTTCTTCA
CTGATGCAAATGCTAGATTTGCAAAGGAACTCATCAGCATACTTGAGGAAGCTGGACAGAAGGTCAGTCC
TGATTTGGCTGCAATGGGGCGTGGTGCACCTCCTCCCCCGTCAGGATAG

(note it would load as XM_024067939.1 the desired name.)

[bradfordcondon]

I can confirm that loading the mRNA file AS IS (without a regexp) resolves the missing features warning.

[kyrenya]

ok! I'm gonna go for it and I'll let you know!
Thanks so much!!

[kyrenya]

could be one of my problems that I was actually using the XXXX_rna_from_genomic.fna file instead of the rna.fna?

[bradfordcondon]

yep.

the file i linked (GCF_002906115.1_CorkOak1.0_rna) has hte headers so they'll be loaded correctly automatically without a regexp:

>XM_024014517.1 PREDICTED: Quercus suber cinA-like protein (LOC112040922), mRNA
ATGTCCTCTCTCCCTCCCCCCGACGCACCATCTTCCTCGACAAGCTCTTTCCCTCCTCCCGAACTCCTCCCCGCCCTGAA
CAAAGTCACGACGCTCCTCAAGGACCGCGGCGAAACGGTCGCCATCGCCGAGACCGCAGCGGGTGGACTGGTCAGCAGTT
CCATCTTGACGGTGGCGGGGGCCTCGCGGATCTATAAGGGGGGCCTGACGCTGTACACGCTGCCGTCGCGCGTCGCGTAT
GCGGGTTGGACAGAGCAGACGATCCGCGACTACCACGGCCCGTCGACGGAGATTGTCGCTGGGCTCGCGAAGCACGTCCG
ACGCGAGCTTGAAAGCACGTACGTCCTCGCTGAATCCGGGACCGCGGGACCCACGGGCGGCGTCACCCCGAATCGGACCC
CGGGGTACGTGGCGTTGGCGGTGGACTGTGACCGGGGCACGTTCGTGAAGGAGCTGGATACCGGGTTGGGCAAGGATAGG
GTGGCGAACATGGTGCGTTTCGCGCAGGAGGCGTTGGAGTTCTTGAGAGAGGTGGTCAGTGGGGATGCGAAATTGTAA
>XM_024014518.1 PREDICTED: Quercus suber alpha-mannosidase At3g26720-like (LOC112041218), mRNA

heres the link again

ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/002/906/115/GCF_002906115.1_CorkOak1.0/GCF_002906115.1_CorkOak1.0_rna.fna.gz

[kyrenya]

everything starts to make a lot of sense... :S

[kyrenya]

ok I used:

Sequence type: mRNA
Method: Insert and Update
Name Match type: Unique Name

Found 65969 sequences, its currently running.. 54% done. I think its going fine. If all goes good, then its just uploading the gff with no extra options, right? as soon its done I will let you know.
Thanks for all the support.

[kyrenya]

uhmm.. my gff upload job is hanged like this:

[kyrenya]

forget it! started uploading, its just it took 10 minutes to show the first line showing the 1.oo % done :)

[bradfordcondon]

yes that GFF file is much slower than others i've tried. I'd assume its because it has so much cross-linking info.

Let me know if the load isnt successful. Otehrwise I'll close this issue as resolved in a couple of days.

/remind me in 48 hours

[reminders[bot]]

@bradfordcondon set a reminder for Nov 14th 2018

[kyrenya]

yes sure, I will let you know! Thanks!

[spficklin]

Thanks @bradfordcondon for jumping in while I was away.

[bradfordcondon]

no problem. @kyrenya any update on your job?

[kyrenya]

still running! I think still needs a couple of days maybe.. its at approx 30%

[spficklin]

Just curious... how long has it been running so far?

[reminders[bot]]

:wave: @bradfordcondon,

[kyrenya]

HI! there's a screenshot:

one day and 3 hours approx, 34%.

[kyrenya]

found this this morning.... had to start over.. any ideas of why this happened?

[bradfordcondon]

You lost connection to the db... normally I'd say this was a broken connection and you should run really long jobs with screen or nohup to prevent this from happening.

Im a little unsure though because your session lost connection to the db, not to the server, so im not positive...

[kyrenya]

uhm.. that is weird because in fact I'm using a tmux session.. so that is exactly my fear, that is something else.. I will keep you posted if it happens again.. but now I'm again in the 11% :(

[bradfordcondon]

ok so its not that.... maybe someone else has an idea, sorry

[kyrenya]

no problem, let's see if I'm lucky and it doesnt happen again.

[bradfordcondon]

after consulting with @almasaeed2010 you are running out of transactions, so the only real remedy for that is to increase the number of allowed transactions or to close the process.

So if it fails again, what I'd suggest is breaking up the GFF into pieces. Sounds like quarters or thirds would do it. You can associate each one with the same analysis.

[kyrenya]

Thank you so much for the ideas! I will let it run again, just to check at which percentage fails again, and to be sure its not going to be uploaded, and then I will try your suggestions and let you know!

[kyrenya]

hi again!
I update you of my gff fight status:

last week, the server were I am installing the database site, crushed. I lost connection to it and it was completely frozen. I didnt related this with the gff upload. I rebooted it, and since I was going to re-run the gff job, I made the following changes in order to not run out of transactions:

I set up:
max_connections = 350
Shared_buffers = 512 MB
Couldnt find the option kernel.shmmax in /etc/sysctl.conf

gff _upload launched again in friday:

yesterday night, my server had crush again..

resuming: I didnt manage to upload the gff yet, we will try to check what is happening to the server, and I'll keep you posted. I'm aware that this is maybe a bit out of topic, but any suggestions are more than welcome! thanks!

[kyrenya]

starting in a new machine.... I will keep you posted once I'm again in gff step...

[spficklin]

Just curious was that most recent error related to the same run you started 6 days before? Or did that one finish and this error is from a different job?

[kyrenya]

no, it didnt finished. but also didnt give error, it just happened that the connection to the remote pc was lost because the machine crashed. We think maybe the machine was not stable for some reason, I'm starting again the tripal site in a machine that is not 'desktop' version, but 'server' version. Did I explain myself? I will keep you updated as soon as I try to upload again the gff.

[kyrenya]

done!! :) :) :)

[kyrenya]

I have a new issue though:
I've just published gene records, and I got this as a result (chose one just to explain the situation)

as you see, the cross reference wont work, I guess because I dont have an accession there.. Is there a way of fixing this without uploading again everything?

Thanks in advance!

[kyrenya]

I solved it adding a URL to the databases GeneID and GeneBank automatically produced by the gff upload. Now I have Cross Reference active links, but I still dont have anything in the Accession field. I've seen that in the in the citrus example also that field is empty..

[bradfordcondon]

The accession field is the primary dbxref, which is an internal accession number. IE if my site is the waffle_database, a gene might be waffle:00001: its internal to my site. This means I wouldn't expect it to be set when loading the GFF.

The feature.dbxref_id field defaults to NULL. i think this field needs to be tweaked so the label doesnt appear if NULL.

Does that make sense? With that, is your issue resolved?

[kyrenya]

ok, so if I understood well, tha accession is the primary dbxref, and its saved in the feature.dbxref_id, and its for my site accession. Dummy question: that will be NULL always, or it will be useful to have an id (like waffle:00001).. when does that happen? yes, maybe we can close this issue, since the gff is already correctly uploaded and we can discuss thid in #765.

Thanks a lot one more time.
Best.

[bradfordcondon]

Dummy question: that will be NULL always, or it will be useful to have an id (like waffle:00001).. when does that happen

it happens when you set it somehow (manualy, sql, a custom module). it could always be null. or you could decide my tripal site is going to have accession numbers, and set them.

[kyrenya]

hi again!
so as you know everything went in the end ok with uploading gff and fastas.
When trying to add blast results data, I got a warning saying that my feature IDs (the XM_####) were ambiguous.. and then is when I have noticed that after uploading gff and fastas, when publishing mRNAs, I got approx the double of records I expected having this:

one being:

the other:

is this because in the gff..

NW_019805567.1 Gnomon mRNA 10441 10998 . - . ID=rna31;Parent=gene24;Dbxref=GeneID:112040922,Genbank:XM_024014517.1;Name=XM_024014517.1;gbkey=mRNA;gene=LOC112040922;model_evidence=Supporting evidence includes similarity to: 1 Protein;product=cinA-like protein;transcript_id=XM_024014517.1
NW_019805567.1 Gnomon exon 10441 10998 . - . ID=id235;Parent=rna31;Dbxref=GeneID:112040922,Genbank:XM_024014517.1;gbkey=mRNA;gene=LOC112040922;product=cinA-like protein;transcript_id=XM_024014517.1

I see the relation here between them:

by the way, only in that one. In the polypeptide there is no relationship found.

So my question is.. should I upload again the gff but choosing a regexp to tell tripal to catch only the XM_#### that is after 'Name=' so that it doesnt pick also all the exons as if they were mRNAs? Is that what is happening?

Thanks in advance,
Cirenia.

[bradfordcondon]

so you have two mrnas with the same name. one identifier (uniquename) is the same as the name, the other is rna31.

furthermore I can see that theone whose identifier is XM_024014517.1 mRNA has no relationships, or other information. All of the info is tied to rna31.

Finally, one mRNA has CDS sequence (rna31), the other has the raw sequence. I beleive that if you use SQL to query your features, you'll find that one has a feature.residues value associated with it(XM_024014517.1), and one doesnt (rna31).

This leads me to believe: the first mRNA, rna31, is the one created by the GFF. everything looks correct.
The second mRNA (XM_024014517.1) was created when you loaded FASTA sequences, and you incorrectly specified that XM_024014517.1 is the uniquename of the feature, when in fact, it is the name of the feature.

When you loaded the FASTA for the mrna, did you specify upate only for the method, or insert and update? If you had checked update only, this shouldnt have happened (or I'm wrong :) )To check if im correct, you could reload the FASTA file for the mRNA using the configuration below, and see if feature with the name (rna31) now has a mRNA sequence in addition to a (CDS) sequence.

[kyrenya]

thanks... I think I uploaded it with those parameters, but since Im not sure because of these duplication issue and your comment, I will upload again. Maybe I use a minified version of the data to check faster. I will keep you posted of my progress!

[kyrenya]

ok, so I did this with a minified version and there are no replicated mRNAs entries anymore!
I'm not sure if what happened is that I chose insert and update when I uploaded the mRNA fasta, or that at some point I picked unique name instead of name.. but with your advice it worked! so now I'll go for the big ones! thx again :)

[kyrenya]

ok, everything run fine when I used the minified data. But now I'm starting over the upload of the gff, the genomic fasta and the mrna fasta, and this is what happens:

in order to not have the replicate mrna entries, I do the following:

load genomic.fasta first:
matching NAME, and INSERT ONLY way of upload.

load mrna.fasta second:
matching NAME, and UPDATE ONLY.

load the gff: here comes the problem...

we have been here before.. this way the gff will take years to be uploaded..
any help in how to do this, and not to get the double entries? I feel im running in circles..

thanks again!

[laceysanderson]

Hi @kyrenya were you ever able to get this GFF3 imported? If not, we've been doing a lot of work on the GFF3 importer in #1040 for maybe it's worth another try?

[kyrenya]

Hi Lacey! I got it indeed, but it was some time ago! Sorry for not updating this issue I really thought I did. My work in CorkOakDB finished, now I am not currently working in databases, so cannot try the new stuff now. But I will take advantage of this post to really thank all the Tripal community for the great support that was of great help for me while building the site. Cheers for that! And best wishes!

[laceysanderson]

I'm glad you got it! Good luck on all your current and future adventures and hopefully we see you make your way back to Tripal one day <3

[kyrenya]

Never closed the door I enjoyed it very much!! Best, Cirenia.

El jue., 15 oct. 2020 16:34, Lacey-Anne Sanderson [e-mail]
escribió:

I'm glad you got it! Good luck on all your current and future adventures
and hopefully we see you make your way back to Tripal one day <3

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tripal/tripal/issues/677#issuecomment-709407374, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AKJ5PJOFAVEVVQO6SWFQOF3SK4I75ANCNFSM4FW4FQZA
.