Suitecrm: Scheduler AOD/LUCENE index problems

[adamjakab]

I'm not sure what these are for and for now I have disabled them but I cannot run cron with these enabled:

php cron.php
PHP Fatal error: Uncaught exception 'Zend_Search_Lucene_Exception' with message 'Index doesn't exists in the specified directory.' in /home/crm2.ingrossomaterialipulizia.it/httpdocs/modules/AOD_Index/Lib/Zend/Search/Lucene.php:566
Stack trace:

0 /home/crm2.ingrossomaterialipulizia.it/httpdocs/modules/AOD_Index/AOD_Index.php(318): Zend_Search_Lucene->__construct('modules/AOD_Ind...')

1 /home/crm2.ingrossomaterialipulizia.it/httpdocs/modules/AOD_Index/AOD_Index.php(68): AOD_Index->getLuceneIndex()

2 /home/crm2.ingrossomaterialipulizia.it/httpdocs/modules/Schedulers/_AddJobsHere.php(678): AOD_Index->optimise()

3 [internal function]: aodOptimiseIndex(Object(SchedulersJob))

4 /home/crm2.ingrossomaterialipulizia.it/httpdocs/modules/SchedulersJobs/SchedulersJob.php(522): call_user_func_array('aodOptimiseInde...', Array)

5 /home/crm2.ingrossomaterialipulizia.it/httpdocs/include/SugarQueue/SugarCronJobs.php(188): SchedulersJob->runJob()

6 /home/crm2.ingrossomaterialipulizia.it/httpdocs/include/SugarQueue/SugarCronJobs.php(228): in /home/crm2.ingrossomaterialipulizia.it/httpdocs/modules/AOD_Index/Lib/Zend/Search/Lucene.php on line 566

[mrbarletta]

Got this issue after an upgrade from a previous version.

Check you have this directory
modules/AOD_Index/Index/Index/
content should be like this:

`-rw-rw-rw- 1 www-data www-data 0 Feb 27 03:00 optimization.lock.file

-rw-rw-rw- 1 www-data www-data 0 Feb 27 15:01 read.lock.file

-rw-rw-rw- 1 www-data www-data 0 Feb 27 03:00 read-lock-processing.lock.file

-rw-rw-rw- 1 root root 43 Feb 27 03:00 segments_1uc

-rwxrwxr-x 1 www-data www-data 20 Feb 27 03:00 segments.gen*

-rw-rw-rw- 1 www-data www-data 0 Feb 27 03:00 write.lock.file
`

In my case I was missing some 0 sized files, i created them and voilà!

[dpavlovic]

I have this log/problem during upgrade procedure (using wizard) 7.6.10 -> 7.7.9.
`26-Jan-2017 05:07:01 America/New_York] PHP Fatal error: Uncaught exception 'Zend_Search_Lucene_Exception' with message 'Index doesn't exists in the specified directory.' in /home/xxxxxxxxxxx/public_html/modules/AOD_Index/Lib/Zend/Search/Lucene.php:566
Stack trace:

0 /home/xxxxxxxxxxx/public_html/modules/AOD_Index/AOD_Index.php(342): Zend_Search_Lucene->__construct('modules/AOD_Ind...')

1 /home/xxxxxxxxxxx/public_html/modules/AOD_Index/AOD_Index.php(83): AOD_Index->getLuceneIndex()

2 /home/xxxxxxxxxxx/public_html/modules/Schedulers/_AddJobsHere.php(683): AOD_Index->optimise()

3 [internal function]: aodOptimiseIndex(Object(SchedulersJob))

4 /home/xxxxxxxxxxx/public_html/modules/SchedulersJobs/SchedulersJob.php(536): call_user_func_array('aodOptimiseInde...', Array)

5 /home/xxxxxxxxxxx/public_html/include/SugarQueue/SugarCronJobs.php(188): SchedulersJob->runJob()

6 /home/xxxxxxxxxxx/public_html/include/SugarQueue/SugarCronJobs.php(228): SugarCronJobs->executeJob(Object(SchedulersJob))

7 /home/xxxxxxxxxxx/public_html/cron.php(74): Sugar in /home/xxxxxxxxxxx/public_html/modules/AOD_Index/Lib/Zend/Search/Lucene.php on line 566`

[pgorod]

Have any of you tried to recreate the Index from scratch? I'm curious if the problems would reappear or not. I would bet these are all just inconsistencies in the Index. They might be Zend bugs, not SuiteCRM bugs.

Although I believe this issue #2788 would need to be solved before we can really acquit SuiteCRM from messy installations found after upgrades... in the current state of things, it's only natural that upgrades will mess things up occasionally, you just need a bit of bad luck.

[pgorod]

@samus-aran based on my experience in the forums, I would say this issue could be closed as it is somewhat common to see broken Indexes that need rebuilding. This could be caused by Lucene bugs, by degraded permissions (#2787), or by unfortunate interactions of cron jobs with upgrade processes (#2788).

It would be useful to have a bug report where the corruption event could be reproduced reliably, but that doesn't seem to be the case with this Issue, and it's been quite inactive for a long time.

Of course this is just a suggestion and if any of the people in the thread object, I have no problem with keeping this open.

[chris001]

@pgorod I respectfully have to not agree with you on this one. This issue is truly a bug which kills the user experience and should stay open until it is fixed.

• The application should not just bomb out on the user like that.
• The application should not expect the user to become an engineer and deal with debugging the code.
• The application should catch the error, tell the user what happened, in plain language, and either explain to the user how to remedy the problem, or better yet, the app should actually fix the problem itself - create a folder that it knows the 3rd party library depends on being there, change permissions of files, whatever is necessary so that "IT JUST WORKS (TM)".

[pgorod]

@chris001 I agree with all you wrote, this needs to work! And SuiteCRM badly needs solving all these tripping points.

What I meant was: this is a frequent problem; as far as I can tell, the causes are probably elsewhere, in other Issues already being dealt with. I think for most cases the corruption comes from wither the cron jobs running as a different user (already mitigated in 7.8.3) or the cron jobs running during maintenance and upgrade (#2788); or maybe there are bugs in Lucene, but it seems we should be dropping Lucene in favor of another library that is still maintained. So...

What I mean is I can't extract anything useful from this particular Issue to help me debug. I would need a reproducible case of the corruption being produced, not just the verification that it has occurred in the past. But I do agree this needs to "just work"!

[chris001]

They're getting the bug when upgrading 7.6 to 7.7.9.

1. Upgrade 7.6 to 7.7.9 and see if bug happens. It should.
2. Upgrade 7.7.9 to current version 7.8.3 and see if bug happens.

[pgorod]

I believe it's much more random than that, it only happens on a few occasions; I think it's an interaction between two processes: the AOD Optimize Index cron job (which is throwing the error) and the installer process.

I'm afraid I don't have time now to go into this sort of investigation... but even if I had, I would prefer to devote it to #2788, I believe it would solve this issue and a bunch of others. :-)

[chris001]

This bug is one which can and should be tested automatically by phpunit tests run by Travis or by the command line. Travis/phpunit/command line is already able to silently install the app. Silent upgrade is practically the same thing.

[jyotiraghav93]

I am also getting the same issue, Uncaught exception 'Zend_Search_Lucene_Exception' , any luck?

[pgorod]

https://pgorod.github.io/Reindex-AOD/

[pgorod]

@Dillon-Brown I would close this issue, even though this code is obviously lacking something, there is nothing in this issue that helps us debug. There's no point in keeping this open forever. Now that we are beginning to have proper tests we should be getting closer to the basic frailties of the indexing mechanism.

[jyotiraghav93]

Asunto: Cron cd /home/webdesi0/public_html/suitecrm; php -f cron.php
Fecha: Thu, 24 May 2018 14:05:02 -0500
De:

Cron Daemon [e-mail]
Para: [email protected]

PHP Fatal error: Uncaught exception 'Zend_Search_Lucene_Exception' in /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Storage/File/Filesystem.php:65
Stack trace:

0 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Storage/Directory/Filesystem.php(359): Zend_Search_Lucene_Storage_File_Filesystem->__construct('modules/AOD_Ind...')

1 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Index/SegmentInfo.php(286): Zend_Search_Lucene_Storage_Directory_Filesystem->getFileObject('_h.cfs')

2 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene.php(495): Zend_Search_Lucene_Index_SegmentInfo->__construct(Object(Zend_Search_Lucene_Storage_Directory_Filesystem), '_h', 1, -1, NULL, 1, true)

3 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene.php(570): Zend_Search_Lucene->_readSegmentsFile()

4 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/AOD_Index.php(342): Zend_Search_Lucene->__construct('modules in /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Storage/File/Filesystem.php on line 65

Fatal error: Uncaught exception 'Zend_Search_Lucene_Exception' in /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Storage/File/Filesystem.php:65
Stack trace:

0 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Storage/Directory/Filesystem.php(359): Zend_Search_Lucene_Storage_File_Filesystem->__construct('modules/AOD_Ind...')

1 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Index/SegmentInfo.php(286): Zend_Search_Lucene_Storage_Directory_Filesystem->getFileObject('_h.cfs')

2 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene.php(495): Zend_Search_Lucene_Index_SegmentInfo->__construct(Object(Zend_Search_Lucene_Storage_Directory_Filesystem), '_h', 1, -1, NULL, 1, true)

3 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene.php(570): Zend_Search_Lucene->_readSegmentsFile()

4 /home/webdesi0/public_html/suitecrm/modules/AOD_Index/AOD_Index.php(342): Zend_Search_Lucene->__construct('modules in /home/webdesi0/public_html/suitecrm/modules/AOD_Index/Lib/Zend/Search/Lucene/Storage/File/Filesystem.php on line 65

this issue is not fixed through your given link https://pgorod.github.io/Reindex-AOD/ already tried it

[pgorod]

Those errors are not the same as the one previously mentioned in the issue. That's why I feel this should be handled in the Forums, not here. Here we need to be able to move beyond individual, disparate user reports; we needs stuff that is reproducible or that, at least, points to some specific area of the code for us to look into. IMHO :-)

[jyotiraghav93]

Okay thanks

[Dillon-Brown]

I'm not able to replicate this issue so I agree we should take this to the forums and treat it as individual user issues until someone can reliably replicate this as a bug that we can investigate.