Autoupgrade fails with: Execution of Oracle Base utility, /u01/app/oracle/product/19c/dbhome_1/bin/orabase, failed for entry upg1.

Yesterday I was upgrading a DB on our production environment and I was facing this error while running the deploy:

Execution of Oracle Base utility, /u01/app/oracle/product/19c/dbhome_1/bin/orabase, failed for entry upg1.

We downloaded the latest autoupgrade version as recommended and used this simple config file:

	hostname: >  $ORACLE_HOME/jdk/bin/java -jar /u01/app/oracle/product/19c/dbhome_1/rdbms/admin/autoupgrade.jar -version
	build.version 22.3.220503
	build.date 2022/05/03 11:55:20 -0400
	build.hash 9e84e228
	build.hash_date 2022/05/03 11:37:00 -0400
	build.supported_target_versions 12.2,18,19,21
	build.type production
	hostname: >
	
	hostname: > cat sample_config.cfg
	global.autoupg_log_dir=/export/home/oracle/logs
	global.target_home=/u01/app/oracle/product/19c/dbhome_1
	global.target_version=19
	#
	# Database number 1
	#
	upg1.dbname=DB_TO_UPGRADE
	upg1.start_time=NOW
	upg1.source_home=/u01/app/oracle/product/18.0.0/dbhome_1
	upg1.sid=DB_TO_UPGRADE
	hostname: > 

Prechecks/Analyse did not have any issues:

	hostname: > java -jar /u01/app/oracle/product/19c/dbhome_1/rdbms/admin/autoupgrade.jar -config sample_config.cfg -mode analyze
	AutoUpgrade 22.3.220503 launched with default internal options
	Processing config file ...
	+--------------------------------+
	| Starting AutoUpgrade execution |
	+--------------------------------+
	1 Non-CDB(s) will be analyzed
	Type 'help' to list console commands
	upg>
	upg> Job 100 completed
	------------------- Final Summary --------------------
	Number of databases            [ 1 ]
	
	Jobs finished                  [1]
	Jobs failed                    [0]
	
	Please check the summary report at:
	/export/home/oracle/logs/cfgtoollogs/upgrade/auto/status/status.html
	/export/home/oracle/logs/cfgtoollogs/upgrade/auto/status/status.log
	hostname: >	

Still, when we tried to run the deploy, we were getting this error:

	hostname: > echo $ORACLE_HOME
	/u01/app/oracle/product/18.0.0/dbhome_1
	hostname: > java -jar /u01/app/oracle/product/19c/dbhome_1/rdbms/admin/autoupgrade.jar -config sample_config.cfg -mode deploy
	AutoUpgrade 22.3.220503 launched with default internal options
	Processing config file ...
	upg1
	Execution of Oracle Base utility, /u01/app/oracle/product/19c/dbhome_1/bin/orabase, failed for entry upg1.  Review the log file /export/home/oracle/logs/cfgtoollogs/upgrade/auto/autoupgrade_user.log for possible causes.
	
	hostname: >

If we check the autoupgrade_error.log, we see a this error:

2022-06-14 12:35:16.401 WARNING upg1
Execution of Oracle Base utility, /u01/app/oracle/product/19c/dbhome_1/bin/orabase, failed for entry upg1.  Review the log file /export/home/oracle/logs/cfgtoollogs/upgrade/auto/autoupgrade_user.log for possible causes.
- Bootstrap.processCLIParams
oracle.upgrade.autoupgrade.utils.errors.ConfigurationError: upg1
Execution of Oracle Base utility, /u01/app/oracle/product/19c/dbhome_1/bin/orabase, failed for entry upg1.  Review the log file /export/home/oracle/logs/cfgtoollogs/upgrade/auto/autoupgrade_user.log for possible causes.

	at oracle.upgrade.autoupgrade.config.links.UserConfigAssembler.evaluateErrors(UserConfigAssembler.java:100)
	at oracle.upgrade.autoupgrade.config.links.UserConfigAssembler.process(UserConfigAssembler.java:69)
	at oracle.upgrade.autoupgrade.config.CLILink.nextLink(CLILink.java:53)
	at oracle.upgrade.autoupgrade.config.links.GlobalParametersValidator.process(GlobalParametersValidator.java:48)
	at oracle.upgrade.autoupgrade.config.CLILink.nextLink(CLILink.java:53)
	at oracle.upgrade.autoupgrade.config.links.StandbyListResolver.process(StandbyListResolver.java:132)
	at oracle.upgrade.autoupgrade.config.CLILink.nextLink(CLILink.java:53)
	at oracle.upgrade.autoupgrade.config.links.InclusionListResolver.process(InclusionListResolver.java:94)
	at oracle.upgrade.autoupgrade.config.CLILink.nextLink(CLILink.java:53)
	at oracle.upgrade.autoupgrade.config.links.SourceTargetEvaluation.process(SourceTargetEvaluation.java:63)

If we check the autoupgrade_user.log, no clear reason:

/export/home/oracle/logs/cfgtoollogs/upgrade/auto/autoupgrade_user.log:

2022-06-14 12:33:42.818 INFO The target_version parameter was updated from 19 to 19.13.0.0.0 due to finding a more accurate value
2022-06-14 12:33:44.911 INFO srvctl command is not available or user does not have permission to execute it
2022-06-14 12:33:46.829 INFO Finished processing dbEntry upg1
2022-06-14 12:33:48.220 INFO Current settings Initialized
2022-06-14 12:33:48.252 INFO Starting
2022-06-14 12:34:28.211 INFO File /export/home/oracle/logs/RBTRAIN/102/prechecks/rbtrain_preupgrade.html not found
2022-06-14 12:34:28.211 INFO File /export/home/oracle/logs/RBTRAIN/102/prechecks/rbtrain_preupgrade.log not found
2022-06-14 12:34:30.431 INFO Job 102 completed
2022-06-14 12:34:30.442 INFO Closing
2022-06-14 12:35:12.055 INFO
build.hash:9e84e228
build.version:22.3.220503
build.date:2022/05/03 11:55:20 -0400
build.max_target_version:21
build.supported_target_versions:12.2,18,19,21
build.type:production

Before someone says: what about the “INFO File /export/home/oracle/logs/RBTRAIN/102/prechecks/rbtrain_preupgrade.log not found” ? First: this message is part of the Analyze, Second: those files exists.

What we did to fix this, is to use an old autoupgrade.jar file we had saved from few months back, and upgrade completed with no issues.

hostname: >  $ORACLE_HOME/jdk/bin/java -jar /u01/app/oracle/product/19c/dbhome_1/rdbms/admin/autoupgrade.jar -version
build.hash 081e3f7
build.version 21.3.211115
build.date 2021/11/15 11:57:54
build.max_target_version 21
build.supported_target_versions 12.2,18,19,21
build.type production

We still don’t know why it was failing, but I’m still investidating and trying to replicate problem on a lab. If I get some answers, will post it here.